ThirdParty Repository

[Manuzor]

We should have a repo where all thirdparty sources are contained therein, so the setup of a new krepel workspace is much easier than it is now.

This would involve creating a CMake script which can be included from the outside so the third-party stuff can be included into an existing project.

We should also think about whether to pre-build binaries, although I think this would become quite large and should be avoided. Still, it might be a good idea. But I guess this is best left for the future.

[pampersrocker]

:+1: :+1: :+1:

The pre-built binaries could be stored outside of the git repository (using a fileserver)

[Manuzor]

Indeed. But I'm not sure how we will manage versioning and stuff that way. I.e. using older versions of ez for older versions of krepel. Having something like apt-get for this would be supreme :smile:

I've played around with CMake a bit and its capabilities to include external CMake projects and I think I've come up with a decent solution. When you have an external CMake-powered project, you can simply include that to your current project and CMake will add it to your VS solution, Makefiles, what have you, also resolving dependencies as usual. This would mean that every developer would have to build the thirdparty stuff locally, but only once per checkout/change in the thirdparty repo.

By the way: I will be against using submodules in any case, until Git finally manages to make them work properly :wink: If we have an external thirdparty repo, it would have to be referenced via environment variables, via CMake variables, or our CMake scripts will simply search for them in close proximity such as ../thirdParty, ../../thirdParty. I think should work well. To ease this process, we could provide a shell script that checks out a standard layout of git repos, or something like that.

[pampersrocker]

Well, thats basically what Unreal does with the Unreal Engine, they provide a batch script, which then calls a C# Exe which will check and download all third party tools. Everything whats checked in into the repo is then the exe file and any sort of checksum data file (e.g. yaml) which is used to check the third party files and re-download them if needed.

I think using C#/Mono for that is an easy task and also works on any platform. Another tool is required to create the data file and upload the dependencies to a file server.

[Manuzor]

What about thirdParty headers and Co? Does unreal check them in directly?

[pampersrocker]

Nope they all come with the dependencies. Everything thats checked in in the repo is their own source code.

You cannot create the solution without the dependencies setup (which is a desired result anyway).

They go event that far, that the setup script creates a post pull hook to rerun the dependencies check after each pull.

[Manuzor]

Sounds good! Can we simply take it from them or is their license too strict?

[pampersrocker]

I think we would need to rewrite quite a chunk of that ourselves because this seems to be pretty hardcoded to the Unreal Repo / Servers and so on.

For the license stuff: It's basically all or nothing:

Can I copy and paste the Unreal Engine code into my own project or engine?

If you use any Unreal Engine code in your product (even just a little), then your entire product is governed by the EULA, and royalties are due.

See https://www.unrealengine.com/faq and https://www.unrealengine.com/eula

So if its possible in any way, it will be cumbersome since we have the agree to the complete eula and stuff.

[Manuzor]

so, that's a clear no then :smiley:

Well, how hard do you think it is to write it ourselves? And do you think it might be easier/harder to implement in python?

[pampersrocker]

I think the complexity is basically the same using C# or Python. This could be done in approx. 2 Days (16 Hours), regarding the client side implementation. Probably another day to get the creation done and tested.

Another aspect is the server side backend implementation. This could be done with an easy SFTP-FileServer. But i would suggest putting a bit more effort and create the backend using a php script, or python webserver, ASP.NET, whatever you prefer and the server supports. It should be able to handle some sort of database like management of the data. Basic commands would to to fetch data, create (and delete for cleanup) versions of files.

[Manuzor]

I wonder... Isn't there some protocol out there already for something like this? I mean, it sounds like your everyday package management system, right? If there was something more standard, it would be preferrable I guess.

[pampersrocker]

I cannot find any, this seems to be a complete leak in all libraries I can find, either you would use scm for that or simply a zip file or whatever, but this in between those two.

Git Fat or Git Annex do similar things but they all are not that simple to use or buggy.

Well then create something yourself for that versioned dependencies, then a python based build system, then the World :smiley:.

[Manuzor]

https://github.com/lab132/PyBake

[Manuzor]

Once it is in a usable state, we will be using PyBake. There will be no ThirdParty repo, hence this issue can be closed.