search

aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.56k stars 371 forks source link

Mirror a https repository requiring authentification #1003

Open jc305697 opened 2 years ago

jc305697 commented 2 years ago

Detailed Description

Hi everyone, I would like to ask if it is possible to add the functionality in aptly to create a mirror of an authentification protected site or if it is not possible if anyone as a workaround solution so I can integrate the packages in aptly ?

Context

At my work we are switching to Proxmox VE for our virtualization stack and we would need to create a local mirror of the enterprise repositories for our internal use so we can be sure that every update was tested on our staging infrastructure before it's deployed in production which wouldn't be possible if every server pulls its updates directly from Proxmox repositories. Somebody was able to use wget --mirror to do this but I would like to use aptly if possible since we already use it to manage our Ubuntu packages update. Link for any person interested : https://forum.proxmox.com/threads/subscription-update-pve-with-no-internet-access.29259/

Also we have the required amount of licences to access and download this content on each of our Proxmox VE just FYI.

I think this would be benefit other users if they need to use a repository that needs authentication like an internal repository or a repository offered by a company

Possible Implementation

I have no idea how aptly downloads the packages, but if it uses wget or something like that maybe each repository could have metadata associated with it for the username and password ? To be honest the password could be in plain text in my case since it only consists of the serverid. I don't have any experience in GO but if I can be of any help let me know.

Your Environment

Our aptly server is running on Ubuntu 20.04

aol-nnov commented 2 years ago

@jc305697 I should note, that there are plenty of authentication methods... (and including all of them into aptly is no fun)

For example, to mirror a repo with a client certificate auth, I successfully used nginx as a proxy that handled authentication for me and aptly was working with that nginx host.

You say ugly? I say - it just works, ha!

Here is the example config for basic auth: https://serverfault.com/a/345244

jc305697 commented 2 years ago

Thanks @aol-nnov , I will have to look into what is the authentification method used and you are right adding all authentification methods would be impraticable and would maybe be out of scope of what aptly should do. Thanks for your response ! :)

ryanammoury commented 2 years ago

Hi, I was looking into this myself, as I was trying to mirror a repo with basic http authentication (Apache AuthType Basic configuration)

What worked was to add user:password in the repository URL when creating the mirror.

aptly mirror create https://user: password@my.repo.url.com etc..