Jenkinsfile, trigger elife-bot downstream updates

[lsh-0]

this is roughly what we want I imagine.

We do a release and then trigger downstream jobs with the new tag version.

The downstream (/dependencies/elife-bot-elifecrossref) needs to accept that value and then update that single dependency.

Can't do that with the way things are setup currently, Giorgio seems to have avoided passing values between pipelines - probably for good reason - but we can't do what we really want to do in isolation. That tag value must be communicated somehow.

[gnott]

Is there anything I can contribute to on this PR @lsh-0?

[lsh-0]

hey @gnott - I know I only did this ~29 days ago but I'm trying to recall what it's context was ... is there a related ticket or issue somewhere?

[gnott]

I think it may be related to issue https://github.com/elifesciences/issues/issues/6980

[lsh-0]

ah ok, I think I'm caught up.

In this case we want this:

1. elife-crossref-xml-generation is successfully tagged + released
2. trigger an elife-bot pipeline that updates this specific dependency
3. elife-bot pipeline opens a new PR with the change, going through CI, etc.

The problems we have are:

1. Pipenv doesn't seem to want to update just a single dependency
2. all of the logic for branching, creating a PR with a title, description etc is tied up in the elifeUpdatePipeline with no way to communicate extra information (like tag).

With Pipenv I think we can settle for 'update everything' for now, which isn't what we really want, and for the elifeUpdatePipeline logic I think I'll take another look at what is going on and see what I can do. It may be possible to do pass information via that updateStep closure.

What I have in this PR is probably pseudocode, I don't recall.

[gnott]

A good summary, it sounds like it to me. In the meantime, I'm updating all the dependencies by running the script in the elife-bot project and I can continue that until something else is tried.

[lsh-0]

by running the script

is this the update-dependencies.sh?

I'm working on something else today but I can make sure this gets some attention Monday and into the new sprint if I'm not done by then (cc @NuclearRedeye). Whatever we come up with ~we should apply it to~ it should be applicable to all of the other cases, like bot-lax+elife-tools, lax+bot-lax, etc.

[gnott]

update-dependencies.sh

Yup, it is working fine so far.

[lsh-0]

some time just freed itself up, I can take a look at this now

[lsh-0]

ok, after much complexity I think I have something that may work.

1. this PR isn't pseudocode, except for perhaps the params - they need fixing and I have a change readied for that.
2. it calls an elife-bot Jenkins job that needs to be tested, but it's a variation on Jenkinsfile.update-elife-tools essentially, skipping the update_python_dependency script that we've been using.
3. I now have a way to update a single dependency (and it's dependencies) via Pipenv that I could possibly incorporate into the update-dependencies.sh script so it would look like: ./update-dependencies.sh elifecrossref 0.11.0

One thing you might need to be careful about is all these little python libraries depending on each other. For example, elifecrossref depends on elife-tools, elife-cleaner and elife-article, elife-cleaner and elife-article also depend on elife-tools, they all depend on beautifulsoup, which depends on lxml (heavy) etc ... it's getting complex. Pipenv and it's flexible ~= specifier should help but I wouldn't be surprised to get a weird dependency problem at some point.

I'm out of time for today but will continue Monday.

update: modified update-dependencies.sh script here: https://github.com/elifesciences/elife-bot/pull/1364

give it a whirl and let me know if it works for you.

[gnott]

The failure of the test pipeline may be related to recently removing coveralls, and there may be something which is still trying to submit a coverage report using coveralls token. It's not specified in this repository, I think, so it might be somewhere in test formulae or in builder configuration.

[lsh-0]

The elife-bot update elifecrossref Jenkinsfile: https://github.com/elifesciences/elife-bot/pull/1366

First successful build (sorry for the email spam it must have been sending you): https://alfred.elifesciences.org/job/dependencies/job/dependencies-elife-bot-update-elifecrossref/5/console

The automatic elife-bot PR to update elifecrossref: https://github.com/elifesciences/elife-bot/pull/1367

The changeset in that PR is a bit weird but that should clear up once the update-dependencies.sh changes are merged.

[lsh-0]

pipenv will need to live outside of this script because it's dependencies are getting locked in as well:

https://github.com/elifesciences/builder-base-formula/pull/322

update: and there is a weird Ubuntu bug about running 'pip freeze' that causes this: https://github.com/pypa/pip/issues/4022 still happening in 18.04 apparently.

[lsh-0]

first green auto PR: https://github.com/elifesciences/elife-bot/pull/1370

not sure why pip and setuptools are now included in the lockfile but ... anyway.

I'll get these PRs merged in and sorted

[lsh-0]

I'm thinking that Jenkinsfile.update-elifecrossref could become general purpose really easily. I should do that, otherwise it just encourages copy+paste ...

[lsh-0]

https://github.com/elifesciences/elife-crossref-xml-generation/pull/126