Change gas-preprocessor submodule to a subtree.

[orbea]

git-submodules are a pain in the ass, especially when they are as large as ffmpeg and when github starts throttling my already slow internet speed. RetroArch has since changed their submodules to subtrees which has worked very well and I would like to do the same for ppsspp. This would help package maintainers and anyone else who routinely clones ppsspp.

Since ffmpeg is the biggest offender I suggest starting there, but changing the gas-preprocessor submodule should probably be the first step.

If this change is acceptable I can make additional PRs for the other submodules next.

Here is some documentation on git-trees. https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844

An more ideal future solution would to bake in the parts of ffmpeg that ppsspp relies on and entirely avoid submodules or subtrees, but that change is more involved and beyond my current ability.

[hrydgard]

I don't understand, how exactly does subtrees make things better? Submodules are easy to manage, just run git submodule update --init --recursive to update them all, recursively.

[orbea]

I may not be the best person to explain all the benefits, but some considerations.

1. Submodules do not work well with shallow clones since its unknown how deep the referenced commit is. See this issue report for more discussion on this issue. https://github.com/rust-lang/rust/pull/24521
2. Users do not have to remember to update the submodules.
3. This could be helpful for ppsspp ports like for libretro.
4. My personal biggest gripe is that my build script does not rely on local mirrors and will clone the ppsspp repo along with required submodules such as ffmpeg, but github 9/10 times will decide to clone one or more of the submodules at 200 KiB/s instead of my regular speed of 2-3 MiB/s which will take a very long time for the ffmpeg submodule.

[Alcaro]

(this probably duplicates orbea's comment.)

With submodule, you have to clone the submodule's history, which is several gigabytes for big repos. With subtree, it's all squashed.

With subtree, you clone with git clone and it's done. With submodule, you need some other command (git submodule init?) as well after cloning, or you'll get arbitrary compiler errors.

With subtree, you update with git pull and it's done. With submodule, you need to remember updating submodules, or risk stuff like this.

[unknownbrackets]

Well, PPSSPP's submodules are not gigabytes. It doesn't seem like this problem has motivated rust to change from submodules - why is that? Are they still in progress, or have they not been convinced?

How does this help libretro?

Most of our submodules don't update often, is your build script doing a clean, unreferenced clone every time? If so, why not keep a periodically updated "base", cp -r it into a workdir, and then update/compile/etc. that? It will be much faster, subtrees or otherwise.

-[Unknown]

[Alcaro]

Well, PPSSPP's submodules are not gigabytes

Judging by this part of OP

especially when they are as large as ffmpeg and when github starts throttling my already slow internet speed

they're big enough to be annoying, gigabytes or not.

How does this help libretro?

We hate submodules. But yeah, that follows from the other arguments in the thread, I see no reason to call it out specifically.

Most of our submodules don't update often

"Most" "not often" so some of them do update often?

If it happens at any point ever, everyone has to pull the updates. And we don't know when they update, so we'll have to either do it every single time, or risk absurd errors we've forgotten how to google.

And updates at any point ever or not, fresh clones still have to deal with them.

is your build script doing a clean, unreferenced clone every time?

Last time I checked, our script just kept one copy around and used git pull. Don't even know if it used git clean properly. (Where "checked" means "saw someone have to manually reset the buildbot because someone force pushed, so git pull choked".) Either way, easy to add git submodule update --sacrifice-virgins=72 to, I think we do that already.

But that doesn't help the humans. orbea isn't a buildbot.

[hrydgard]

Not convinced so far. I really don't want all of ffmpeg's code inside the main PPSSPP repo, so I'm closing, sorry.