Epic: Mirror third-party dependencies downloads

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.28k stars 408 forks source link

Epic: Mirror third-party dependencies downloads #4281

Open bayandin opened 1 year ago

bayandin commented 1 year ago

Motivation

If we can't download a package (for example for extension) due to third-party problems, we can't build a compute-node image, which is bad.

Ref:

DoD

We can build a compute image from scratch if all uncontrolled by us resources are unavailable (this includes git repos, which could be deleted)

Implementation ideas

Upload files to S3
Fork git repos to neondatabase account (or create a separate org for this)

Tasks

Other related tasks and Epics

jamesbroadhead commented 5 months ago

I support a separate GH org for forks

bayandin commented 5 months ago

I support a separate GH org for forks

Sounds nice, we can even add patches if required (for example for pgvector).

The only downside I can see is that if, for some reason or by mistake, GitHub decide to block any of the repos, this affects all the forks, and we won't be able to build the image. So we might be

And, to not to forget in the future: We download postgis from their host, the tarball comes with autogen.sh, if we switch it to the build from a source, then we need to add autoconf/automake to the image (and run autoreconf).

andreasscherbaum commented 4 months ago

We need to consider how and when to merge changes from upstream into our fork/repo/copy. Also how notifications about patches and releases reach us. Not every project does timely releases, especially not for bugfixes.

The idea with additional patches is good, we can show what we change when building extensions. That enables transparency.

One idea to work around the blocking problem is to run a separate GitLab instance instead.