search

rabbitmq / erlang-debian-package

Fork of the official Debian package for Erlang
https://launchpad.net/~rabbitmq/+archive/ubuntu/rabbitmq-erlang
Apache License 2.0
12 stars 9 forks source link

Files not available during new builds ? #1

Closed rockandska closed 5 years ago

rockandska commented 5 years ago

Hi,

Not sure if it is directly related to your workflow or to bintray, but it seems that some builds are triggered and that during that builds some files are delete.

I just saw this happened and saw directories appeared / disappeared on bintray repository. As a result, deployment / CI failed with : 

Failed to update apt cache: E:The repository 'https://dl.bintray.com/rabbitmq/debian stretch Release' does no longer have a Release file.

When you aware of, you could check that this kind of directory bellow exist to know that a build is on its way, but "standard" users will just concluded with "damn, why it is doesn't work anymore ?"

.work_1550237646669/

It will be better if the files could be replaced at the end of the build to minimize the indisposition time.

Regards,

michaelklishin commented 5 years ago

The Release and other repo metadata files are produced and updated by Bintray, our pipeline simply publishes a new version using their public API.

rockandska commented 5 years ago

Thanks for you answer, but could you please report them this behaviour since you are customers ? I would like to report them directly but i don't find an easy way to exchange with them directly ( no slask, no irc, )

michaelklishin commented 5 years ago

We do have one extra communication channel besides what any other Bintray user has but it really should be used for major issues. I don't think this issue is in any way specific to our repositories, so please report it directly.

rockandska commented 5 years ago

I've spent may times trying to explain to Bintray guys the fact that every time you push a new release that it put the repository in an unstable state and redo all the packages, but even with the screenhosts bellow, no one seems to understand....

https://imgur.com/a/uT0IfNt

I hit this behaviour, on 15 February, on 25 February and i'm pretty sure you pushed a release each time.

If you wanna taking over this issue i've opened to Bintray, the ticket is the #99124, but i think i will switch to Cloud Package.

Regards,

michaelklishin commented 5 years ago

@rockandska there was a release on February 15th but there were no releases on February 25th. This includes Erlang packages which Concourse produces automatically as soon as it detects a new tag.

I don't know for how long this "transitionary state" of reindexing lasts but in order to hit it your systems must check for new versions at least every hour. Can you elaborate on how you deploy RabbitMQ packages and why?

We will contact Bintray support to clarify how they perform reindexing when a new version is published but chances are we won't be significantly more convincing. It can be a major architectural change for them and only affect more complex (multi-arch, sophisticated metadata) package types such as Debian.

rockandska commented 5 years ago

@rockandska there was a release on February 15th but there were no releases on February 25th. This includes Erlang packages which Concourse produces automatically as soon as it detects a new tag.

Could you please take a look at the release history for rabbitmq,elixir,etc.. ?
Because, from what i've seen, the reindexing is full ( rabbitmq, erlang, elexir etc..) and not only on the package pushed, so I think a new release of a package was pushed on 25 February. (if you check the timestamp of the files, you will see the same for all files from my tests)

I don't know for how long this "transitionary state" of reindexing lasts but in order to hit it your systems must check for new versions at least every hour. Can you elaborate on how you deploy RabbitMQ packages and why?

This transition state seems to let (at least for erlang ) the repository unstable for around 15min the last one I hit this behaviour. I hit it, because i validated my role to install Erlang with molecule (destroy,creation of new instance at each validation) a few times on 15 February. The screenshot on 25 February was taken when i wrote to Bintray, i was just checking the repository and i've saw their ".workdir_XXX", so i've waited to see if the behaviour will happen, and it happened, so i took the screenshots to elaborate what i've said to Bintray.

We will contact Bintray support to clarify how they perform reindexing when a new version is published but chances are we won't be significantly more convincing. It can be a major architectural change for them and only affect more complex (multi-arch, sophisticated metadata) package types such as Debian.

Sure, but right now, i don't know if the reason is my English is not good enough, but they seems to not understand the behavior described and why it is a real problem when you have some CI who relay on external services as Bintray who could fail every time a new release is pushed on the Bintray account

michaelklishin commented 5 years ago

Erlang and Elixir use a separate org now. The original Debian repo hasn't seen any releases of those two since November 2018.

RabbitMQ Debian package on Bintray says it was released on February 28th. That doesn't match the change log (which I update after we produce a release) or the latest run of our 3.7.x GA release job (which I triggered for 3.7.11 and 3.7.12). So either the date is off or some other event at Bintray triggered the reindexing.

In any case, we do not control how/when Bintray reindexes packages and why. But I also don't recall more than two reports of this, which to me suggests that to hit it one has to run provisioning often enough as reindexing happens relatively quickly (of course, I have no monitoring or data to back this guess).

Feel free to switch to Package Cloud if you hit it, that's why we have two alternatives for package repositories.

michaelklishin commented 5 years ago

@rockandska your English is perfectly fine :) We have some background with JFrog/Bintray and should be able to explain what we the outsiders see.

rockandska commented 5 years ago

Just to let you know @michaelklishin

Thank you for your reply. With regard to the issue you were having, we have managed to reproduce this issue. We are escalating this ticket to our Bintray Developers in order for them to research this. We will update you as soon as we get an answer from them. Best regards, JFrog Support

michaelklishin commented 5 years ago

@rockandska oh, that's good to know. Thank you very much for sticking with it and sharing the progress!