fix: don't delete existing content on the File Share and fix the destination folder

jenkins-infra / crawler

tools crawler

26 stars 65 forks source link

fix: don't delete existing content on the File Share and fix the destination folder #130

Closed lemeurherve closed 12 months ago

lemeurherve commented 12 months ago

This PR:

fixes the File Share synchronization as currently this job is erasing the content uploaded to our update center prototype by our update-center2 test job.
fixes the folder sync to ensure the content is in the correct destination folder, extracting the SAS token query string from the existing credentials to its own credentials so we can set the destination folder.
logs each crawler name before executing them.

Note: while aws-cli creates the folder if needed, azcopy fails with a 404 error: I had to manually create the updates folder in the File Share.

Ref:

https://github.com/jenkins-infra/helpdesk/issues/2649

daniel-beck commented 12 months ago

Note that this should not delete existing output from the target directory even. https://updates.jenkins.io/updates/ notice anything about the file modification times? Some crawlers don't work, but the existing files will ensure continued operation of the plugins relying of them. They'll just slowly become outdated.

dduportal commented 12 months ago

Note that this should not delete existing output from the target directory even. https://updates.jenkins.io/updates/ notice anything about the file modification times? Some crawlers don't work, but the existing files will ensure continued operation of the plugins relying of them. They'll just slowly become outdated.

absolutely! the deletion only happened in the "new and not used" targets added in #128. The rule of thumb we have is "DO NOT CHANGE THE CURRENT updates.jenkins.io SYSTEM AND PROCESS" => so we don't plan to ever touch the rsync deployment step at all.

dduportal commented 12 months ago

Moving back to draft until @lemeurherve finished his tests

lemeurherve commented 12 months ago

Some crawlers don't work, but the existing files will ensure continued operation of the plugins relying of them. They'll just slowly become outdated.

Thanks for the info!

I imagine we'll have to ensure we copy them over the File Share and the R2 buckets too.

Should we open issues for each of them? (FlywayInstaller & PerlInstaller)

lemeurherve commented 12 months ago

Moving back to draft until @lemeurherve finished his tests

Test successful, ready for review.