Create new flare-data repo

[clintbellanger]

I plan on creating a new flare-data repo to separate the core engine from the current data. This will help projects like Polymorphable with keeping their engine in sync with this one.

Here's how it affects the Flare folders:

• art_src will be permanently moved to the new repo. A copy of the logo file should be placed into the distribution folder.
• All mods subfolders besides default will be permanently moved to the new repo.
• tiled will be permanently moved to the new repo.

I'll create the flare-data repo. If anyone knows the exact commands I can use to create this new repo while preserving the change history of art_src, mods, and tiled, please let me know.

[stefanbeller]

I did not try it myself, but stackoverflow usually yields very good (correct) answers. http://stackoverflow.com/questions/3910412/split-large-git-repository-into-many-smaller-ones http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository

[AMDmi3]

This, althrough slow, seems to do the thing:

git clone <flare repo> && cd flare
git filter-branch --tree-filter 'find . -not -name art_src -not -name mods -not -name tiled -mindepth 1 -maxdepth 1 -exec rm -rf {} \;' --prune-empty

there seem to be a faster variant with git filter-branch --index-filter, but that'll require writing an extra script to filter git ls-files output and passing it to git rm --cached

Also, although this removes commits that only touch removed files (e.g. commits which will be empty after filter), but it doesn't remove empty merge commits, which should be done as they pollute history a lot.

This gives some hints: https://www.google.ru/search?q=git+remove+empty+merges

[makrohn]

With either of those methods, just be prepared to walk away for half an hour or an hour and go get lunch, then, depending on your upload speeds, to go get a second lunch.

I probably would do something similar to this method here, including a second cloned repo: http://dound.com/2009/04/git-forever-remove-files-or-folders-from-history/ Not necessarily the RIGHT way.

So make a new repo, Flare-Data

$git clone https://github.com/clintbellanger/flare.git engineonly
$git clone https://github.com/clintbellanger/flare.git dataonly
$cd engineonly
$git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch art_src mods tiled" HEAD
$rm -rf .git/refs/original/             //This is the normal filesystem command, not the git rm command.
$git reflog expire --all &&  git gc --aggressive --prune
$git push
$cd ../dataonly
$git remote add flaredata https://github.com/clintbellanger/flare-data.git
$git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch src distribution CMakeLists.txt Launch\ F*" HEAD
$rm -rf .git/refs/original/             //This is the normal filesystem command, not the git rm command.
$git reflog expire --all &&  git gc --aggressive --prune
$git push origin flaredata
$cd ..
$rm dataonly
$git clone https://github.com/clintbellanger/flare-data.git

Something like that would remove all of the history of the other project.

IDEA: The other possibility I'd put forward is to have flare-data be a direct fork or branch of flare-engine, so that someone can download the .tar of flare-data and have everything, src/, mods/, etc, all in a single tarball - essentially, all full conversion games would then be like Polymorphable, a fork+mods, and do regular pull requests from flare-engine into flare-data to keep the src up to speed.

[makrohn]

Also, it should be noted, I'm pretty sure it's VERY important when using --prune-empty in ANY of these methods to make sure you remove all the files in one filter-branch, or some very weird things start happening with misattributed commits.

[clintbellanger]

@makrohn's idea is an interesting one -- I could treat "flare the game" as a direct fork, just as any other major mod would. Thoughts for/against this idea?

Because I have tags for previous versions of Flare in this repo, should I refrain from pruning the art_src, tiled, mods folders here? Would that mean anyone who forks from this repo will always get the entire history of e.g. the fantasycore files (very large image/music files)?

[stefanbeller]

I agree on @makrohn's idea as well, so the flare the game would be the repository just as of now, but flare the engine is a repository having only the spare default mod/config. This makes it easy to start a new fork and getting the latest updates from the engine.

It is also beneficial to flare the game, as there is no overhead with multiple repositories for the user/newcomer. Only one repository to clone and you're done.

[clintbellanger]

Now I'm essentially sold on the idea of creating a flare-game fork.

The remaining question: do we just delete the flare game-data from the engine repo, or do we try to prune it? If we don't prune, anyone cloning the Flare engine gets all the data history (very large image and music files). If we do prune, I'm not sure how it would affect existing tags/branches/clones.

Relevant warning from the man page for git:

WARNING! The rewritten history will have different object names for all the objects and will not converge with the original branch. You will not be able to easily push and distribute the rewritten branch on top of the original branch. Please do not use this command if you do not know the full implications, and avoid using it anyway, if a simple single commit would suffice to fix your problem.

[makrohn]

I personally vote for prune, so that anyone cloning doesn't get the data history, which is one of the major benefits.

But, pruning DEFINITELY means that anyone with an existing clone needs to delete and reclone, everything I've seen makes a point of mentioning that you need to alert all the collaborators.

I would also recommend using a lot of dry-run options and checking out the git logs before commiting and pushing anything.

[clintbellanger]

Would that mean Polymorphable would need to delete and reclone too? Wouldn't that destroy your change history?

[makrohn]

Poly would also need to delete and reclone. Essentially what we'd do would most likely look something like this:

Flare-engine gets split Polymorphable gets pruned down to /tiled and /mods Polymorphable sets flare-engine as an upstream and merges it in Polymorphable cherry-picks our VERY few src/ divergences back in (maybe a half-dozen commits) Polymorphable looks exactly the same as it does today with a history that still has all of it's own commit history and objects, all of the flare-engine history, and none of the cruft

Anyone with an existing polymorphable repo clone deletes and reclones. Right now this might consist of just Thane, Justin, and myself, for all I know.

Polymorphable's other option is to Do an actual fork of flare-engine (we'll call it makrohn/flare-engine for now) (this is the major advantage to this method) Prune pennomi/polymorphable down to /mods /tiled and our few src/ divergences Set pennomi/polymorphable as a remote repo for makrohn/flare-engine Merge the now-pruned pennomi/polymorphable into makrohn/flare-engine Rename makrohn/flare-engine to makrohn/polymorphable Add a note to the pennomi/polymorphable repo that no more updates will be coming from there, and encourage to reclone into makrohn/polymorphable (this is the major disadvantage to this method) Polymorphable looks exactly the same as it does today with a history that still has all of it's own commit history and objects, all of the flare-engine history, and none of the cruft. But then I feel guilty for taking the repo away from Thane.

tl;dr @clintbellanger Yes, but we have easy ways to get it back. And we shouldn't stand in the way of Progress, anyways.

[pennomi]

@makrohn Don't worry about taking the repo away from me; I think Polymorphable belongs to you more than to me after all the great work you've done on it. :)

On the other hand, it might confuse people who currently have that checked out, or people who have an old link to the repo.

All that aside, as @makrohn eloquently sums it up:

we shouldn't stand in the way of Progress, anyways.

[clintbellanger]

I created a flare-game repo that uses flare as its upstream: https://github.com/clintbellanger/flare-game

If we prune "flare" (the engine repo), flare-game will have to take similar steps to polymorphable if we want to maintain the data history. Note, though, the game-data history isn't a big concern for me at all, not compared to the source code. If I ended up having to sacrifice the history of the art files, it wouldn't hurt my feelings.

[makrohn]

Similar steps, yes. Also, probably worth cloning flare.git to a local ~/flarebackup.

[clintbellanger]

I'll attempt the git filter-branch commands and try pushing them to a test GitHub repo.

I'm thinking about creating a separate flare-engine repo, so that downstream clones are obviously obsolete.

[makrohn]

That has DEFINITELY occured to me as an advantage, and then upload a note to the current flare repo "repo obsolete, please go to clintbellanger/flare-engine.git", which is similar to what I plan on putting into pennomi/polymorphable.

[dorkster]

Yes, I think we should look at this as creating two new repos, instead of replacing one (current flare) and adding another (flare data). It'll give us a "clean" start, and we can keep the old repo as a record for anyone looking to play with older versions.

[makrohn]

I wonder if there's a way to migrate issues.

[dorkster]

Hmm, I didn't even think about that. Seeing as issues are a Github specific thing, my guess is leaning towards 'no'.

[stefanbeller]

@dorkster Do not underestimate github ;) We can ask the github staff to migrate the issues or we can try to write a script for it (There is a nice and very functional github api for scripting)

[stefanbeller]

Just read the whole thread here, and I agree that we should do 2 new repos, so in the end there are 3: This one here having just a readme saying: 'outdated, see the other 2 repos' The flare-game repository will have all history preserved (including art), but the flare-engine should definitely be pruned, as nobody cares about the old art, when starting a new game project anyways.

we shouldn't stand in the way of Progress, anyways.

As long as the progress goes the same direction as we're all going... let's not run in circles :P

[AMDmi3]

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch src distribution CMakeLists.txt Launch\ F*"

Should be careful with this and alike, as it'll leave other files/directories which existed in the repo at some point in history (check out first revision, for example - it has quite different directory layout), so "remove everything except" approach is more reliable and correct than "remove this an that". Unfortunately, it's harder to do with index-filter.

[makrohn]

I also forgot to throw a --purge-empty in there.

[clintbellanger]

For the new flare-engine repo:

Ok. So I'm looking for a filter-branch command (or series of commands) that keeps src, distribution, and CMakeLists.txt intact and purges everything else (even things that used to exist, like itemdb). We can ignore the Launch* files (I'll probably drop them) and the mods/default folder for now (the history of that folder isn't important).

If necessary, I don't mind using a command that filters everything but the "src" folder (then adding back a history-less CMakeLists.txt and distribution). The src history is the chief concern for the engine repo.

[clintbellanger]

Perhaps I just need to run filter-branch several times. Is there a way to show all files/folders that existed at any point in the repo's history?

[makrohn]

Trying to work through this with Clint in chat, I did

$git log --pretty=format: --name-only --diff-filter=A | sort - >> list.txt

To get a list of every file that has ever existed in Flare. This allows:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch art_src mods/320x240 mods/frontier mods/fantasycore mods/hud_alt mods/averguard mods/wipcore mods/minicore mods/living_bones tiled art config enemies engine font images items itemdb languages maps music npcs powers quests soundfx tilesetdefs animations" --prune-empty -- --all

[pennomi]

Notice you have "mords/hud_alt" in there

edit: Or not... maybe you've fixed that?

[makrohn]

yeah, I fixed it!

[clintbellanger]

I'm running the command locally too. Once I think I have it correct I'll push to a new github repo for verification.

[clintbellanger]

My attempt: https://github.com/clintbellanger/flare-engine

It looks quite cleaned up -- a fresh clone of it takes up about 3MB. Note that branch and tag data is cleared. Not sure if that was avoidable.

Take a look, see if it's acceptable. If it looks good, polymorphable and flare-game will be created and set to use flare-engine as the new upstream.

[clintbellanger]

Here's a log of what I did.

Create flare-engine

git clone https://github.com/clintbellanger/flare.git flare-engine

List all folders that ever existed in the repo:

git log --pretty=format: --name-only --diff-filter=A | sort - >> list.txt

Use this to make a specific filter-branch command

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch art_src mods/320x240 mods/frontier mods/fantasycore mods/hud_alt mods/averguard mods/wipcore mods/minicore mods/living_bones tiled art config enemies engine font images items itemdb languages maps music npcs powers quests soundfx tilesetdefs animations fonts" --prune-empty -- --all

Cleanup

git reflog expire --all
git gc --aggressive --prune

New origin (create a blank flare-engine repo on GitHub)

(create a blank flare-engine repo on GitHub)
git remote rm origin
git remote add origin https://github.com/clintbellanger/flare-engine.git
git push -u origin master    

That's all for flare-engine.

Next, I did a similar process for flare-game, only I used this command for filter-branch to remove the source code.

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch distribution mods/default src CMakeLists.txt COPYING Launch* REAMDE" --prune-empty -- --all

After completing all of the above steps for flare-game, I pushed the pruned repo to GitHub.

Then:

git remote add upstream https://github.com/clintbellanger/flare-engine.git
git fetch upstream
git merge upstream/master

The only conflict was mods/mods.txt (they're necessarily different). I made sure we're using the flare-game mods.txt. Then I sent the merged flare-game to GitHub.

git push

[clintbellanger]

Next questions:

1. Is it actually possible to re-create the older tags for Flare inside flare-game? When I merged flare-engine into flare-game, am I getting old commits in chronological order?
2. What to do about the Issues? Should we just re-enter the relevant ones on the new repos?
3. What to do with this obsolete "flare" repo? Obviously we can leave it around for now for downloading and for the Issues archive. Is there a way to make a github project read-only? Should we delete this repo at some point in the future?

[pennomi]

@clintbellanger The commits should be in chronological order, if I understand anything about Git. (But don't quote me on that ;) ).

The issues can be migrated manually as necessary. It might be a good time to prune out really old ones we don't want for now.

I suggest leaving the obsolete old repo where it is, and change the readme to clearly say so. Some people may want to go back there for historical reasons?

[dorkster]

This part of the git documentation might be helpful for retagging. I think you're manually going to have to find the proper commits to tag, since the checksums are different than the original flare repo.

And for issues, yes, manually re-adding sounds like the best course of action. Some issues were engine related, and some were game data related, so those will need to be added to their corresponding repos.

[makrohn]

For what it's worth, https://github.com/makrohn/polymorphable.git is now a meager 55.17 MiB in size! This is awesome! We're also clean of all the flare-data commits. I think this is going to be awesomely huge for new total conversions.

What I did to make that happen:

$git log --pretty=format: --name-only --diff-filter=A | sort - >> list.txt
$git clone https://github.com/pennomi/polymorphable.git
$git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch art_src/characters art_src/icons art_src/loot art_src/menus art_src/spells art_src/textures art_src/tilesets mods/320x240 mods/frontier mods/fantasycore mods/hud_alt mods/averguard mods/wipcore mods/minicore mods/living_bones fonts art config enemies engine font images items itemdb languages maps music npcs powers quests soundfx tilesetdefs animations Launch\ F* src tiled/averguard_academy.tmx tiled/averguard_atrium.tmx tiled/averguard_complex.tmx tiled/averguard_plains.tmx tiled/averguard_prison.tmx tiled/averguard_temple.tmx tiled/dungeon tiled/frontier tiled/grassland tiled/grassland_trees.png tiled/cave/ mods/core_gui" --prune-empty --tag-name-filter cat -- --all

Note in the above command, I was taking quite a bit of care to prune away a bunch of non-Poly mods and tiled files. I may have missed one or two, but the impact shouldn't be massive.

$rm -rf .git/refs/original/
$git reflog expire --all
$git gc --aggressive --prune
$git remote rm origin
$git remote rm upstream
$git remote add origin http://https://github.com/makrohn/flare-engine.git
$git fetch origin
$git merge origin/master
$git remote add old https://github.com/pennomi/polymorphable
$git fetch old

Then, I compared a few files using diff (regular diff, not git diff) to find the divergences, and git cherry-pick'd them (called by commit hash) and resolved a couple bugs. Finally,

$git push http://https://github.com/makrohn/flare-engine.git

And then I renamed the repo on github to Polymorphable, rm'd the local directory, and recloned from github. Voila!

There's a few duplicate commit (because they had different hashes), but I'm not going to fret them going forward. I should be able to run pulls through github to get the latest Engine changes, and the repo size is down by 85%!

[makrohn]

"When I merged flare-engine into flare-game, am I getting old commits in chronological order?"

Yes. This is one of the weirdnesses of, and as I understand, one of the few gripes people have about git (from lookin' round the internet). Old commits always go in as their original time. Why is this weird?

1:00 PM Clintbellanger edits source code in his fork. 2:00 PM Makrohn edits a mod in his fork, without knowledge of Clinbellanger's code change. 3:00 PM Makrohn merges Clintbellanger's commit into his own fork. 3:01 PM Makrohn edits his mod to not break with Clintbellanger's latest commit.

Makrohn's repository was actually nonfunctional for only one minute (from folding in Clint's work at 3:00 until the fix at 3:01), but the history shows that he wrote his mod at 2:00 knowing full well that Clint had edited the source code at 1:00, and the game was broken for an hour. Now, the Merge Commit at 3:00 should clear that up, but it's not necessarily immediately obvious if there's another fifty commits in the meantime.

[igorko]

Can I ask why do we need src folder in both repos? I would like to have only source in one repo and only data in another.

[makrohn]

Earlier in the discussion, the point was brought up to treat flare-game like any other full conversion mod, with src in it's repo. One of the advantages is that anyone new can download a single tarball and start playing to game. Anyone wanting to create a total conversion mod can just fork flare-engine and start modding. Given that the src, it's objects, and it's history, are about 1.5 MiB, it's not a lot of overhead. Then, Polymorphable and Flare game (and any other full conversions) will just do regular pulls in from flare-engine. 4th, 6th, 7th, 8th comments. The more full conversions, the better, I think!

[igorko]

I guess default mod doesn't need tooltips.txt. Also I think we should add config files syntax definitions to flare-engine README, or at least add link to flare-game. Because if moder gets engine repo, he will not know from where to start.

[stefanbeller]

I've cloned both of them now and it seems to work fine. Though on the other hand I'd more like it if flare-game would be a real fork of flare-engine, so it would show up in the network view of flare-engine. I am not sure if that is possible here at github.

I like it the way it is now. (The problems will arise later anyways :D)

[makrohn]

The workaround appears to be to have someone else (or an organization) fork flare-engine, rename it to flare-game, and reassign it back to Clint.

[clintbellanger]

Another reason flare-game might want a copy of the source: we may add features into the game that only make sense for that game, and don't really belong in the main engine. Just like any other Flare-based game might handle things.

[dorkster]

@igorko As a matter of fact, tooltips.txt should be in the default mod. The joysticks listbox in the config menu uses tooltips, since joystick names are usually longer than the width of the listbox itself.

But yes, we should add base versions of config files that would be used in a full game. Although, that would make merging more difficult. Maybe it should be part of the README, or even the wiki. Speaking of which, thats another thing that needs to be moved over from the old repo: the wiki.

[stefanbeller]

We can close this?