search

aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.58k stars 376 forks source link

Aptly in build pipeline #996

Open umerkk opened 3 years ago

umerkk commented 3 years ago

Hi

We are looking at using Aptly in our build pipeline to create & update the packages on the S3 hosted Repo. Currently, we do not have any repo, so I was thinking are there are recommendations of how to do that?

What we are planning is: Preplanning: Publish a dummy Repo on S3.

Build Pipeline: Create a mirror of dummy s3 repo. Create Snapshot of the mirrored repo. Add the newly build package. Publish the new snapshot to S3.

Does this seem like a right approach to do? given we do not want to have a master server that host local repo or something like that, but we would like to push changes to the repo while we are building a new version of our product?

r4co0n commented 3 years ago

You can create your repository locally using aptly's repo module, and then publish snapshots of that repository to S3. aptly's S3-support is restricted to publish items, afaik.

I hope you have seen https://www.aptly.info/doc/overview/, it should help you along.

r4co0n commented 3 years ago

An S3 endpoint for the pool would be quite interesting, it would contain almost everything to make other publications just fancy signed forwarders.

umerkk commented 3 years ago

I could create a repo & publish to S3, but what I am trying to achieve is automation during our CD process instead of a central VM containing the aptly db with all snapshots & repo metadata.

So, if I create a repo & publish in a CD process, the next time, I wonder how it'd understand that there is a history of the S3 repo (previous snapshots etc). I was thinking of uploading the aptly DB into a centralized place & having the CD pipeline download & use that db. It'll helo the runner have consistent repo state each time its building.

nooga commented 2 years ago

Hey @umerkk. Did you succeed? I'm trying to achieve the same but with no luck so far.

umerkk commented 2 years ago

@nooga we ended up using jfrog artifactory for this since the feature sets were not fully flushed.

r4co0n commented 2 years ago

I was thinking of uploading the aptly DB into a centralized place & having the CD pipeline download & use that db. It'll helo the runner have consistent repo state each time its building.

You might want to look at the work being done in PR https://github.com/aptly-dev/aptly/pull/1023, if I'm understanding you correctly this should fix this particular part of the issue by externalizing the database using etcd allowing multiple aptly instances to access it simultaneously.

neolynx commented 1 month ago

PR #1023 was merged...

maybe this help solving the problem as suggested ?