Github large file warnings.

bakercp commented 9 years ago

This issue is just to acknowledge that we are aware of the large static lib file sizes in our package and are working to fix it with the apothecary scripts. So, if you see warnings like:

remote: warning: GH001: Large files detected.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File addons/ofxAssimpModelLoader/libs/assimp/lib/ios/assimp.a is 52.57 MB; this is larger than GitHub's recommended maximum file size of 50 MB

when you're pushing to github, we're working on it ...

bilderbuchi commented 9 years ago

:+1:, thanks for flagging this. :-)

danoli3 commented 9 years ago

Also github has a hard limit of 100MB for individual files in pushes. It will fully reject the push if it has a file greater than that size now.

Although github has a solution for this: integrating and using a new git standard of LFS: https://git-lfs.github.com

bakercp commented 9 years ago

I got an invitation to LFS. Haven't tried it yet, and not exactly sure how it would solve this problem ... but it would be super nice if we could start pushing all static libs over there instead of to the main repo. Does the oF org have access to LFS yet?

pizthewiz commented 9 years ago

The monthly LFS Data bandwidth afforded by GitHub seems to be quite low so far and might be tied to the user/org billing tier. That said, we could use git-lfs and host the files ourselves still…

Each $5USD / month LFS data pack boosts the limits 50GB storage and 50GB bandwidth.

danoli3 commented 9 years ago

Yeah you can point git LFS to a custom server for the files. The current github LFS bandwidth is the issue. I am unsure why they have set it so low for now, we would run out of bandwidth with 3 downloads...

So potentially the statics could be hosted on http://lfs.openframeworks.cc/0.9.0/libraries/. If we paired this with cloud flare or some free cloud service, should take the pressure off the single server.

I am also on the beta. I'll have a look how easy this is to setup.

kylemcdonald commented 9 years ago

@danoli3 let us know what you discover! :) hosting is not an issue as rackspace has graciously offered free cloud servers for OF.

pizthewiz commented 9 years ago

Yep, we could run the reference Git LFS API server ourselves or write our own API client if we already had some sort of dynamic web app we wanted to build on.

Not sure what kind of limits you guys are seeing, but my personal account is running the small plan and LFS is limited to 1GB of storage and 1GB of bandwidth a month. It is possible that the limits are higher for organizations, I'm really not sure but who ever runs the org might be able to request access and find out: https://github.com/early_access/git-lfs

bilderbuchi commented 9 years ago

Yeah, LFS looks potentially nice, but

The Github bandwidth offers are really strange (why 1:1 storage:traffic?), and definitely not fitting for our usecase, so it would definitely be rackspace for us.
The LFS test sever is clearly marked as not production-ready, so I don't think we should use that (yet). Also, I think we shouldn't cook up anything ourselves, cause we would have to maintain that, and it would lead to even more tool fragmentation.
I am a bit concerned about vendor lock-in here, since this would essentially tie us to Github's way of dealing with large binaries (there are several others available). OTOH, Github, being huge, maybe have a chance to get their standard included into git proper. I am not aware of any motions in that direction (Why oh why?), and for some reason they decided against improving/contributing to one of the tools already available, which makes me strongly think that they want to monetize this more heavily in the future. In the end, it's xkcd standards all over again, which makes me kinda mad/sad.

Aside of that, please continue the discussion in #1805, which is the relevant issue for this topic.

OP:

This issue is just to acknowledge that we are aware of the large static lib file sizes in our package and are working to fix it with the apothecary scripts.

danoli3 commented 8 years ago

Just adding to this, Git LFS went 1.0 and now is embedded in Github.

Github Announcement: https://github.com/blog/2069-git-large-file-storage-v1-0

The protocol is found in the github desktop app pre-installed, however the official git install is here: https://git-lfs.github.com

This looks like our solution.

pizthewiz commented 8 years ago

This looks like our solution.

We would need to purchase several "data packs" as users and organizations are allocated 1 GB of storage and 1 GB of bandwidth monthly…

danoli3 commented 8 years ago

You can host your own server https://github.com/github/lfs-test-server

pizthewiz commented 8 years ago

Ahh nice, perhaps worth an evaluation post 0.9.0 to see what the deployment and administration story is like on the Rackspace machines for a Go application. Bit of a bummer that it doesn't support password-less or ssh key use yet.

drlight-code commented 8 years ago

One could think about having a deployement target in our new cmake build ;) which just builds dependant libs, zips them up and uploads them to the web server where there is already bandwidth available. What do you think?

openframeworks / openFrameworks

Github large file warnings. #3833