search

OSGeo / proj-datumgrid

Historic repository for proj datum grids. New developments are at https://github.com/OSGeo/PROJ-data
40 stars 33 forks source link

git-lfs is not cost free #57

Closed rouault closed 4 years ago

rouault commented 4 years ago

@hobu @kbevers I've issued a benign pull request which fails with https://travis-ci.com/OSGeo/proj-datumgrid/builds/130801548

Error downloading object: world/egm08_25.gtx (c18f20d): Smudge error: Error downloading world/egm08_25.gtx (c18f20d1fe88616e3497a3eff993227371e1d9acc76f96253e8d84b475bbe6bf): batch response: This repository is over its data quota. Purchase more data packs to restore access.

Quick googling for that error message leads to https://github.com/nabla-c0d3/nassl/issues/17

Going to https://github.com/organizations/OSGeo/settings/billing , I see we are at "167% bandwidth, 14% storage used ". I'm not sure if every Travis build or "git clone" for folks count for bandwidth usage, but if so then it would be 150 MB each time, so with 7 downloads a month we are out of bandwidth. Apparently for 5 USD / month, storage and bandwidth quota go to 50 GB.

Apparently gitlab.com offers 10 GB LFS storage and unlimited bandwidth : https://about.gitlab.com/2015/04/08/gitlab-dot-com-storage-limit-raised-to-10gb-per-repo/

What should we do ?

kbevers commented 4 years ago

Well, that's a bummer. I guess someone's gotta pay the pills.

What should we do ?

I don't see myself maintaining this repository on GitLab Maybe ask for OSGeo funds? Or community sponsorships in some form? Or maybe we should just go back to the old school way of dealing with the grids on download.osgeo.org/proj/ ...

rouault commented 4 years ago

I don't see myself maintaining this repository on GitLab

migrating it or maintaining it ? Either shouldn't be that much a problem (I would volunteer to do the GitLab migration if needed. Most of the work would be to actually migrate the CI, but it should remain simple enough to do). This is more the act of moving that is a bit annoying, but this repository is probably not followed very actively, so that shouldn't disrupt too much

ask for OSGeo funds?

The issue I see is that there's still a limited bandwidth. This is something we cannot control, so some script kiddie (or someone doing a lot of trials&errors with a pull request - a bit surprised we didn't hit that before !) can easily put us out of bandwidth even if we pay for 50 GB.

hobu commented 4 years ago

From https://help.github.com/en/articles/about-billing-for-git-large-file-storage

Additional storage and bandwidth is offered in a single data pack. One data pack costs $5 per month, and provides a monthly quota of 50 GB for bandwidth and 50 GB for storage. You can purchase as many data packs as you need. For example, if you need 150 GB of storage, you'd buy three data packs.

Apparently for 5 USD / month, storage and bandwidth quota go to 50 GB.

I believe it is 50gb/$5 increments. I pay for git-lfs support for PDAL, and we never hit the 50gb of traffic (it is for our sample data). The biggest bandwidth hog we should be careful about is Travis. I say we ask OSGeo for it.

rouault commented 4 years ago

The biggest bandwidth hog we should be careful about is Travis

We could actually prevent this with https://docs.travis-ci.com/user/customizing-the-build/#git-lfs-skip-smudge since we don't create tarballs from Travis. I did a quick try in a test repository and it works: the LFS links are not resolved. There would be just a test in world/CMakeLists.txt to adapt a bit for that situation.

kbevers commented 4 years ago

migrating it or maintaining it ? Either shouldn't be that much a problem

Maintaining it. I have no other business on GitLab and don't want to introduce yet another platform I have to regularly stop by.

I say we ask OSGeo for it.

+1

rouault commented 4 years ago

GDAL PSC agreed to fund the cost, and @msmitherdc just bought the "extra pack" for LFS. So closing that ticket