Decouple assets from binaries

IvanSanchez commented 9 years ago

I think the game assets should be completely decoupled from the codebase. Right now it's a bit of a downer to see all the openscad files being re-compiled every time I do amake clean.

Ideally, SNIS should be able to switch between sets of game assets (models, textures, shaders, materials, sounds, commodities, factions) easily. The current way of overriding the asset directory looks good enough for me.

I suggest following the example of Artemis with VesselData.xml - create some kind of plain text file listing all available models&textures&materials, parse it, dinamically load assets from there, and have the exhaust ports in plain text rather than .h.

For what I've seen, snis_client.c needs some .h files from the models to know where the exhaust ports are. I guess this should be one major bit of code to clean up and a good place to start.

Bonus points given for:

A graphical interface to choose available assets whenever a server is started.
Make the game server (or even the lobby) the only piece that can choose the assets, and have all clients sync the assets from the server they're connected to.
Provide more than one set of assets.
Making separate Debian packages (e.g. snis-core, snis-assets-default and snis-assets-startrek).
Splitting the Git repo structure into code and assets.

DusteDdk commented 9 years ago

Excellent Ivan, where's the patches ?

IvanSanchez commented 9 years ago

I spent all day working on #49, gimme a rest! :-)

DusteDdk commented 9 years ago

Haha that's pretty cool! If you'd write a few lines about how to build the .deb, I could set contigrator ( http://contigrator.wizznic.org/jobs/space-nerds-in-space/ ) up to also build debs ? :)

IvanSanchez commented 9 years ago

I'll leave some instructions on #49.

smcameron commented 9 years ago

There's a mostly-clean target to avoid the openscad recompile issue (mostly).

Not that this effort isn't welcome, thanks. It may take me a little while to get to this, as I've just started a new job. I may separate some stuff out into separate commits -- I already saw some stuff in #49 that needs separating out, I think.

smcameron commented 9 years ago

For what I've seen, snis_client.c needs some .h files from the models to know where the exhaust ports are. I guess this should be one major bit of code to clean up and a good place to start.

This part at least is now fixed by commit 6626177ce6743c12274ea87aa4332c8e9e7b717a. The .h files are still generated in the same way, but they are not built into the binary, instead, they are just read as data files when the game starts up.

MCMic commented 5 years ago

As discussed by email, for me a good first step towards a clean solution on this would be:

Have in this git repo all needed assets to play the game with default solarsystem
Have in a second repo or an archive (and your webserver+assets.txt system) the assets for additionnal solarsystems and missions.

But having outdated assets in the git as is the current situation from what I understood seems wrong.

This solution can easily be achieved by running make update-assets followed by git commit -a to update assets in the git, and then remove all assets listed in the git from the archive and web server to remove the overlap.

Note that if additional assets have sources they should be made available somewhere as well.

smcameron commented 5 years ago

But having outdated assets in the git as is the current situation from what I understood seems wrong.

Only given certain assumptions some packaging system that I don't even use seems to be making.

it seems wrong that a packaging system I don't use should impose constraints on what's in my github repo.

Updating the assets in github still amounts to adding a bunch more binary data into git, which I'd rather not do.

And I want to have all the assets on the website, not just a subset that excludes what's in the github repo.

The github repo contains a set that are sufficient to run. The website contains those assets, plus updated versions of those, plus entirely new assets. That is by design, as it imposes the least amount of damage on top of the damage already done to the github repo by having those assets in there in the first place.

Feel free to fork it if you think it should be otherwise.

smcameron commented 5 years ago

Alright.... here's an idea.

Instead of a single asset directory which the code rummages through, we have two asset directories, one of which can live entirely outside the installed codebase.

The code will search the 2nd, new directory first, and only if it does not find what it is looking for there does it search the 1st, original directory.

That way, users can install your package, which will contain only the assets that are in the github repo. Then they can separately install the supplemental assets from spacenerdsinspace.com, and put them whereever they like -- in their home directory -- don't need to be "installed", and point the game at that directory to supersede the "original" assets.

How does something like that sound?

Thing is, I'm not putting more assets into the github repo or updating any assets in the github repo.

smcameron commented 5 years ago

Hmm, thinking about this a bit more, there is the problem that there do exist things in share/snis which are not on spacenerdsinspace.com and are not assets, but refer to assets (e.g. all the little config files in share/snis/ like ship_types.txt

So the problem I am facing is I don't want to touch any of the png files in the github repo (because they make it too big) and I also want to be able to update them without touching them in the git repo (hence why I came up with the snis_update_assets.sh script to update things from the website.) However, my idea of possibly checking two directories for every file is problematic, because there isn't only one place in the code that would need to be changed, and it's just kind of awkward to make it work that way, and makes debugging harder if you don't necessarily know where files are coming from, can mask bugs, etc. I already have the SNIS_ASSET_DIR environment variable that I can use to override the location that assets come from entirely, however this needs in addition to the files that are on the website, all the little config files like ship_types.txt as well, and those are currently not on spacenerdsinspace.com (though I suppose they could be). Hmm, need to think about it some more.

MCMic commented 5 years ago

it seems wrong that a packaging system I don't use should impose constraints on what's in my github repo.

The packaging system can be adapted to about anything, it’s just it’s a pain to keep up to date when things are complicated.

This is the first game I’ve seen which keeps outdated data within the git repo on purpose, but if it’s the way it’s gonna be I’ll find a way to package it anyway.

Your 2 directories solution seems overkill, and means duplicated version of assets gets installed. This is why I thought it was more natural to split only default solarsystem from other ones, as solarsystems seemed to live in separate directories anyway.

smcameron commented 5 years ago

So I was thinking about this some more, and 2 things occurred to me. If there are files on spacenerdsinspace.com that are updated versions of assets on github, then wouldn't they show up as changes when I did git diff? And that would be problematic for me, and yet that doesn't happen to me, so I guess there must not be such files.

So I decided to check.

$ git clone git@github.com:smcameron/space-nerds-in-space.git
$ cd space-nerds-in-space/
$ make check-assets
[... many lines of output omitted ...]
238 files already up to date
0 files would be updated
488 new files would be created
0 update failures

So, all the files that exist on both sites are identical. The only differences are new files that exist only on the website.

It was my original intention to allow updated files on the website, but I never actually did update any because it would undoubtedly cause me problems by accidentally getting such diffs incorporated into my usual patch stream.

MCMic commented 5 years ago

Ok, that is good news.

I did a PKGBUILD which adds the assets from the archive after the make install: https://gist.github.com/MCMic/357622b3aabf29b1f5596436788cc95d It does work and produce a package weighing 507Mio.

I proposed it to the maintainer, I’ll see if he accepts or if I have to provide it under a different name. Either way this is what I will use for snislive soon.

smcameron commented 5 years ago

Cool.

MCMic commented 5 years ago

@smcameron (In response to https://github.com/smcameron/space-nerds-in-space/issues/161#issuecomment-461055631 )

What about git-lfs: https://git-lfs.github.com/ ? This seems to do exactly what we need (store the binary elsewhere but track it in git)

smcameron commented 5 years ago

If I understand it correctly, Github only gives you 1Gb / month storage and 1Gb / month bandwidth for free. I already have 500Gb of assets. So I could support 2 clones / month before getting charged. See: https://help.github.com/articles/about-storage-and-bandwidth-usage/

I already have 1Tb / month bandwidth on digitalocean.

MCMic commented 5 years ago

I thought we could use git-lfs with https://spacenerdsinspace.com/ as the assets server. But it’s not clear to me how to easily set up a server https://github.com/git-lfs/git-lfs/wiki/Implementations

smcameron commented 5 years ago

I'd rather not set up anything that involves user accounts of any sort.

MCMic commented 5 years ago

git-annex seems way simpler and easier to adapt to snis usecase, and there are rsync and web remote support: https://git-annex.branchable.com/special_remotes/

MCMic commented 5 years ago

https://git-annex.branchable.com/tips/using_the_web_as_a_special_remote/ I think I’m gonna try a few things locally to see what happens if I just add the files from https://spacenerdsinspace.com/ as remote url through git annex.

smcameron commented 5 years ago

Using my replacement asset system, I got this working...

newship

smcameron / space-nerds-in-space

Decouple assets from binaries #50