Better imports definition (registry.xml, fips.yml, etc...)

[floooh]

There should not be a central registry, instead git URLs should be more flexible:

With the short form 'floooh/oryol', fips should assume that the project is on github, otherwise the URL is expected to be a full git URL.

[floooh]

Also allow a '#' to define a git branch or tag, e.g.:

./fips clone floooh/oryol#fips

./fips clone https://github.com/floooh/oryol.git#fips

./fips clone git@github.com:floooh/oryol.git#fips

[mgerhardy]

It would be very nice to have an option to download the code from an url - and only have the yml files in some global git repo on a per-lib-basis. That way it would be a lot easier to update versions. See the MXE way to handle this.

You could for example reference an official release by just giving the official download url for e.g. a tar archive and let. If that would be possible, if would be sufficient to provide a yml file with that url, and a CMakeLists.txt to build the extracted source.

All you have to do to update something to a newer version would then be to update the url.

Repository example: fipsrepo.git

  \- zlib
     \- fips.yml (with URL to official download side)
     \- CMakeLists.txt (to build the extracted archive)
  \- simpleai
     \- fips.yml (git repo - but with the ability to override the CMakeLists.txt to use the fips related cmake file)
     \- CMakeLists.fips.txt

Libs are now able to provide their own fips support. I don't think this is possible right now if they already use CMake as build system, no?

[floooh]

Hmm that's an interesting idea... at least with the libs I have 'fipsified' so far this should be possible. For instance in fips-glfw (https://github.com/floooh/fips-glfw) I'm basically ignoring the glfw CMakeLists.txt files and add my own fips-compatible CMakeLists.txt file in the root directory, and additionally a glfw_config.h file which is normally created by the original glfw cmake files. So it is the original glfw projects in a git submodule, plus a few fips specific files in the project root directory...

The central fips repo would hold those fips-files, plus the information where the original git repo is, and the fips-import process would put the stuff together (not much more complicated then it is now I guess).

At least for simple external which compile to a single library this should be possible.

[code-disaster]

Some more ideas to throw in:

• A way to supersede a module for a project, e.g. to allow cloning a dependency of a fips module from a different Git repository. Say I want to compile fips-hello-world, but use a different fips-hello-dep2 repository.
• When there's an option to specify a Git branch/tag, an exact commit hash might be useful too.
• An easier way to reference dependencies on the local file system, with relative paths to the project folder. It works already with file:///... Git repositories, but that's somewhat inconvenient.

[floooh]

I think the next step would be a more elaborate way to define an import:

imports:
    fips-hello-dep:      # name of the import, will be the directory name
        # free choice of git URL:
        git:     https://github.com/floooh/bla.git
        # branch, tag or revision (SHA-1, depdending on version control system)
        branch/tag/revision:
        # just point to a local path, don't do any remote fetching
        path:
        # optional other ways to fetch remote files
        svn / hg / tar... : 
        # do name lookup through registry:
        name:

[floooh]

I have started to implement this, currently only git: and branch: is implemented (branch: can be a branch or a tag). Directly git-cloning an untagged revision doesn't seem as trivial as 'git clone --branch x', so I didn't implement this yet. The old format still works as expected and gives an 'obsolete' warning.

Example from Oryol's fips.yml:

imports:
    gliml:
        git: https://github.com/floooh/gliml.git
    fips-glm:
        git: https://github.com/floooh/fips-glm.git
    fips-unittestpp:
        git: https://github.com/floooh/fips-unittestpp.git
    fips-zlib:
        git: https://github.com/floooh/fips-zlib.git
    fips-glfw:
        git: https://github.com/floooh/fips-glfw.git
    fips-libcurl:
        git: https://github.com/floooh/fips-libcurl.git

[fungos]

+1 for this. I had tried Rust a little and I can see fips becoming something like Rust's cargo (check (http://doc.crates.io/guide.html). Cargo is really amazing and helps so much that I hope fips can become the equivalent for C/C++. For this is important to have centralized registry, something like what @mgerhardy said but not obligatory. The way fips works with fips.yml is already very good but we need a way to override this locally (cargo does this: http://doc.crates.io/guide.html#overriding-dependencies), this would work nice for fips. As for a central repository, I think you should create an organization "fips" and then http://github.com/fips/registry could be it where we do pull requests to add things. Also fipsified projects could have aliases, so we can do something like "fips clone glfw" instead of using fips-glfw.

OT a bit, I've fipsified recast/detour but only tested on linux. The demo works too, but I'm pretty sure the macos and windows specifics aren't working: https://github.com/fungos/fips-recast

[floooh]

The ability to override 2nd level imports of an imported module is a nice feature, I'll open a separate ticket for this.

[floooh]

Hi @fungos, I did a quick check of fips-recast on Windows and Mac, all the static libs seem to compile fine, the only problem I found is in RecastDemo that SDL.h can't be found (I don't have that installed). A fipsified SDL would be nice indeed :)

[floooh]

@fungos: Oh wait, I did have to do a change in the CMakeLists.txt file. The set_target_properties here:

https://github.com/fungos/fips-recast/blob/master/CMakeLists.txt#L117

...needs to go below the fips_end_app(), since the target is actually only created in fips_end_app(), and cmake can't set a property on a non-existing target.

[mgerhardy]

I tried SDL once - but it has so many dependencies on its own that I stopped at some point (See their CMakeLists.txt for more details - it's a lot of stuff)

[code-disaster]

I did a build config for SDL in a different build system once. My goal was to do a "minimal" compile, as a static library, for Windows, Linux, OS X (all OpenGL), Android and Raspberry Pi (GLES). It's been a major pain to setup, esp. to work around their own autoconfigure/make system.

[fungos]

@floooh thanks, I will fix that. About SDL, I think it will be easier to port RecastDemo to glfw than get a fipsified SDL :)

[floooh]

some overlap with this thread: #69

[fungos]

Up :) Getting back updating some fipsified repos, but due the fear of breaking oryol or anything else that may happen to use some of my repos, I'm back to check if the feature of importing a repository at a specifc commit is at all supported/used?

Something like:

    fips-imgui:
        git:    https://github.com/fungos/fips-imgui.git@59d83bf

this may do a git clone + git checkout 59d83bf for example. If it is not supported yet I may take a look at it, but to be usable it would be nice to have some command to lock the trees (again, as rust's cargo system uses the toml.lock for these things). So this may require a new file instead of adding the @59d83bf to fips.yml, something like fips.lock with contents like:

fips-imgui: 59d83bfd74c3b46d5f6183c66b9c63208f6c62ac

[floooh]

Hi fungos, currently only checking out a named branch or tag is supported (see here, search for 'branch:' http://floooh.github.io/fips/imports.html).

I agree that we should look at how cargo does things. I think checking out a specific revision by version hash is more important for now then having a lock file. When I did the branch/tag name feature I haven't found a simple way to clone a repo to a specific version, only directly clone a specific branch/tag (see here: https://github.com/floooh/fips/blob/master/mod/tools/git.py#L37), but I might not have looked hard enough I guess ;)

A clone plus checkout to specific version would work I but it may collide with the 'shallow clone' of only 10 versions back to reduce checkout size (would be interesting to see how cargo solves this). If both things are not possible at the same time, then better do a full clone (but only if a specific revision is requested?)

[fungos]

Yeah, git does not support that and cargo does a full clone + checkout.

The problem with tag is that we (lib users) haven't control over tags, so they aren't really useful when importing projects that doesn't have a well defined tagging policy. I can see that a full clone may be overkill in some cases, but I do hope that bigger projects will have a well defined tagging policy.

As there is no supported way of doing a shallow clone at specific ref-hash, we can do the full clone only if there is a ref-has specified.

This means that the lib user will decide the best way for his own project.

Full cloning may happen to not be a problem at all in most projects like fipsified ones. They are tiny and normally have submodules that is already a full git clone anyway?

[floooh]

True, the biggest chunks are usually the actual project submodules, so a full clone + checkout might not be too bad in most cases.

[fungos]

Ignoring some personal opinions and Go specifics, this blog post has some interesting observations in the package management domain: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

[floooh]

Good read, I actually would prefer to wrap an external, proper dependency solution instead of creating our own (the same way fips wraps cmake). I was considering Peru (https://github.com/buildinspace/peru) but didn't have the time/need yet to look into it.

[fungos]

I agree! Better than duplicate the functionality, peru seems to be a nice fit for fips! I may be able to help with it, what is your planning for this integration?

A point that maybe is still valid in the post IMHO is the ability of adding and removing dependencies by command instead of manipulating the fips.yml manually. It would be nice just to type: "fips import oryol" and have it looking at registry.yml then adding this to fips.yml (or peru's) imports section:

    oryol:
        git: https://github.com/floooh/oryol.git

This followed by a "fips reup" (wrap over peru reup) for example should make it really easy to manage imports.

[floooh]

• with Peru, I'm still in the 'pre-evaluation' stage, I haven't even played around with it yet, and don't know how it would fit into fips
• I'm not a big fan of the registry.xml anymore, I think it is better to always directly use repository URLs, the only advantage of the registry is non-technical, by providing a list of fipsified projects, but even that is not complete anymore, and could be done by a simple web site or project wiki page

[fungos]

Forgot to comment about the registry.

Yes,I agree with the fact that registry.xml is not an ideal solution, mostly because it is in the fips repo and requires commits to change.

A site would be better, the problem with a site is the maintenance, hosting, dev, etc. So, unless fips grow enough to demand a full-blown site we should find a very simple solution.

One idea would be to just use a short cut to github like: ./fips import flooh/oryol. It should continue to work with full urls normally.

For other features like the import command, lock files and everything else would depend on the peru evaluation from this point onward. :)

I may give it a try or even try to do a little PoC with fips. I haven't though much about how to import it, just some thinking on how to adapt it to our needs.

UPDATE:

TL;DR: Tried peru (master) locally, couldn't get it to work after some time.

I've started changing fips fetch to convert fips.yml to a fips.lock (a valid peru.yml) and integrate peru as a module for fips. This required some fixes on fips to use python3.3 and update to pyYaml plus some modules that peru uses.

My idea was to quickly use fips to generate peru.yml simulating a lock file, as peru will modify the yml to add revision information. The problem with this is that we must also read peru.yml to update it in the case we add a new import to fips.yml without losing any revision info that peru wrote there, so this idea may require a better thinking/state management. The same for the case of implementing a "import" verb for managing fips.yml. Another problem is that pyYaml does not keep comments when loading/saving, so manipulating yml is a damaging operation.