search

mg5amcnlo / mg5amcnlo

Other
64 stars 35 forks source link

Server Update #16

Closed oliviermattelaer closed 1 year ago

oliviermattelaer commented 2 years ago

Milan

vmadgraph (new louvain)

UIUC (going to disapear)

kratsg commented 2 years ago

So it looks like UIUC is going away (/cc @msneubauer @matthewfeickert @mhance). Do you know what happens to all of the tarballs of the old versions? I can't seem to find them right now and I suspect that's part of the server maintenance mentioned here, but any ETA on when they will reappear if at all? Currently a blocker for our CI in mapyde that we use to build a couple of docker images.

matthewfeickert commented 2 years ago

@kratsg I don't have any good advice for you here unfortunately as the Docker images I've been building have used the http://madgraph.phys.ucl.ac.be mirrors to pull down the MG5aMC_PY8_interface.

I've only used the Illinois mirror for publicly viewing in my web browser given that it seems the MadGraph mirrors don't share public viewing permissions across mirrors.

Can you share what components from the Illinois mirror were critical that you can't get from the other mirrors?

kratsg commented 2 years ago

The launchpad doesn't really contain all the tarballs necessary. For example, for 2.6.3.2, this URL used to work: http://madgraph.physics.illinois.edu/Downloads/MG5_aMC_v2.6.3.2.tar.gz -- but now I can't find this tarball anywhere.

oliviermattelaer commented 2 years ago

So it looks like UIUC is going away

I have indeed set uiuc in redirecting mode trough our new milan server and uiuc will be decomissioned in a couple of months (so at that time no-redirection will be in place anymore creating potential issue for old version of MG5aMC which will still do request to that server).

The launchpad doesn't really contain all the tarballs necessary. For example, for 2.6.3.2, this URL used to work: http://madgraph.physics.illinois.edu/Downloads/MG5_aMC_v2.6.3.2.tar.gz -- but now I can't find this tarball anywhere.

2.6.3.2 should not be used any more it is so/too old. It also had 4 debug version (i.e. the last bug fix for 2.6 feature is 2.6.7) and the tarball of 2.6.7 is available on launchpad (still extremelly old but already much better than 2.6.3.2). I would strongly suggest to use tarball of the LTS version (currently 2.9.x). Those tarball will stay on launchpad forever (but I also suggest to keep up with the last digit of the LTS since they correspond to bug fixing).

kratsg commented 2 years ago

2.6.3.2 should not be used any more it is so/too old. It also had 4 debug version (i.e. the last bug fix for 2.6 feature is 2.6.7) and the tarball of 2.6.7 is available on launchpad (still extremelly old but already much better than 2.6.3.2).

We need to have access to versions like these for exact reproducibility of published papers. Can we have the tarballs back instead of being removed? We're not using 2.6.3.2 for active work but it is used for verifying/checking prior results. The tarball did exist. Why can't it just continue to exist?

matthewfeickert commented 2 years ago

Can we have the tarballs back instead of being removed? We're not using 2.6.3.2 for active work but it is used for verifying/checking prior results. The tarball did exist. Why can't it just continue to exist?

This is one of the reasons that I've been asking about having public tags on GitHub in Issue #10. Being able to have reproducible analyses is critical. Assuming all of the old tarballs haven't been deleted forever, it would be great to have these be publicly available in perpetuity if the tags aren't going to be ported over to GitHub.

oliviermattelaer commented 2 years ago

You have all version2 version tag in bazaar here: https://code.launchpad.net/~madteam/mg5amcnlo/series2.0 and for version3: https://code.launchpad.net/~madteam/mg5amcnlo/3.x

matthewfeickert commented 2 years ago

You have all version2 version tag in bazaar here: https://code.launchpad.net/~madteam/mg5amcnlo/series2.0

Where is the public tag information / how do you access the tags from a public API?

Unless I'm (hopefully) wrong bzr can't do things like provide tag information in the version control information

https://stackoverflow.com/questions/15024597/how-to-download-a-bazaar-repository-with-revision-history-without-using-the-bzr

Poking around https://bazaar.launchpad.net/~madteam/mg5amcnlo/series2.0/changes and assuming "pass to 2.6.3.2" means "this is the commit for v2.6.3.2" then I think(?) revision 279 is the correct revision.

There's no tarball available as the "download tarball" link is broken so attempting to access this with bzr

$ bzr branch lp:mg5amcnlo/2.0
$ cd 2.0
$ bzr update -r279
$ cat VERSION 
version = 2.6.3.2
date = 2018-06-22

So I guess this is the recommended workflow.

oliviermattelaer commented 2 years ago

Looks like the last update of the launchpad website, broke the "download tarball". So yes for the moment, one is force to use bzr. With that version, you can then you can create the official tarball via the command:

./bin/create_release.py

(and likely accept all the warning that it will raise, including(?) the de-activation of the auto-update feature)

kratsg commented 2 years ago

Note: a lot of people (myself included) relied on something like this

http://madgraph.physics.illinois.edu/Downloads/MG5_aMC_v${MG_VERSION}.tar.gz

and this is a pretty silent breaking change that should be noisier to let others know.

oliviermattelaer commented 2 years ago

Well, madgraph.physics.illinois.edu is costing 2k dollar per year to keep the address/machine up. So we decide that it was not worth the cost to maintain that website (Tim has been paying that cost for year). So yes pointing on that is certainly not a long term option. (for the moment we do have forward on place for the most important file but this will disappear soon).

Note that all official public link for the tarball should be pointing on launchpad website since ~10 years. (including the link from the Downloads page on madgraph.physics.illinois.edu).

kratsg commented 2 years ago

Cost is a fair justification, but there currently is no alternative replacement to download the 2.6.3.2 tarball example I provided before. In fact, even going through bzr is a lot more hoops and hurdles for just getting it working because one needs to dig out the corresponding revision and hope it's the one matching the tarball one expected from the old website (no way to confirm via checksum, contents, etc... as the tarball is just gone now).

kratsg commented 2 years ago

(Also, I'm sure CERN would be happy to host all the tarballs until the eventual heat death of the universe...)

matthewfeickert commented 2 years ago

So I've done the following 

$ docker pull python:2.7
$ docker run --rm -ti python:2.7 /bin/bash
root@2b3fbea79846:~# cat /requirements.txt 
epydoc==3.0.1
matplotlib==2.2.5
six==1.14.0
root@2b3fbea79846:~# apt update && \
apt install -y bzr && \
python -m pip install --upgrade pip setuptools wheel && \
python -m pip install --requirement /requirements.txt && \
bzr branch lp:mg5amcnlo/2.0 && \
cd 2.0 && \
bzr update -r279 && \
./bin/create_release.py
Updated to revision 279 of branch /2.0

version = 2.6.3.2

WARNING: The release time information is : date = 2018-06-22

Do you want to continue anyway? (y/n)y
find 304 for the revision number -> starting point for the auto-update
version on the web is 971
CRITICAL: current version on the web is 971
This FORBIDS any auto-update for this version.
Do you want to continue anyway? (y/n)y
Branching /2.0 to directory MG5_aMC_v2_6_3_2
Branched 304 revisions.                                                                                                                                                               
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| In /2.0/MG5_aMC_v2_6_3_2/madgraph/various/plot_djrs.py:
| Import failed (but source code parsing was successful).
|     Error: IOError: [Errno 2] No such file or directory: './Events/run_01/tag_1_djrs.dat' (line 66)
|   
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| In /2.0/MG5_aMC_v2_6_3_2/madgraph/madevent/hel_recycle.py:
| Import failed (but source code parsing was successful).
|     Error: SyntaxError: invalid syntax (hel_recycle.py, line 91)
|   
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| In /2.0/MG5_aMC_v2_6_3_2/aloha/aloha_parsers.py:
| Source code parsing failed (but introspection was successful).
|     Error: Error during parsing: invalid syntax (/2.0/MG5_aMC_v2_6_3_2/aloha/aloha_parsers.py, line 37) -- Bad dotted name
|   
Warning: No information available for madgraph.madweight.mod_file.TestMod_file's base unittest.TestCase
Warning: No information available for madgraph.madweight.mod_file.TestMod_file's base unittest.TestCase
Warning: 88 markup errors were found while processing docstrings.  Use the verbose switch (-v) to display markup errors.
mkdir: created directory 'download-temp'
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data.  See "bzr help launchpad-login".
Branched 167 revisions.                                                                                                                                                               
--2022-07-15 16:11:31--  http://collier.hepforge.org/collier-latest.tar.gz
Resolving collier.hepforge.org (collier.hepforge.org)... 129.234.186.186
Connecting to collier.hepforge.org (collier.hepforge.org)|129.234.186.186|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://collier.hepforge.org/collier-latest.tar.gz [following]
--2022-07-15 16:11:32--  https://collier.hepforge.org/collier-latest.tar.gz
Connecting to collier.hepforge.org (collier.hepforge.org)|129.234.186.186|:443... connected.
ERROR: The certificate of ‘collier.hepforge.org’ is not trusted.
ERROR: The certificate of ‘collier.hepforge.org’ has expired.
--2022-07-15 16:11:32--  https://bitbucket.org/peraro/ninja/downloads/ninja-latest.tar.gz
Resolving bitbucket.org (bitbucket.org)... 104.192.141.1, 2406:da00:ff00::22c0:3470, 2406:da00:ff00::22cd:e0db, ...
Connecting to bitbucket.org (bitbucket.org)|104.192.141.1|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-07-15 16:11:33 ERROR 404: Not Found.

Create the tar file MG5_aMC_v2.6.3.2.tar.gz
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: no default secret key: No secret key
gpg: signing failed: No secret key
Running tests on directory MG5_aMC_v2_6_3_2
['INSTALL', 'doc.tgz', 'PLUGIN', 'doc', 'models', 'madgraph', 'HELAS', 'vendor', '.bzrignore', 'MadSpin', 'LICENSE', 'README', 'proc_card.dat', 'tests', 'UpdateNotes.txt', 'bin', 'input', 'aloha', 'Template', 'mg5decay', 'VERSION']
Traceback (most recent call last):
  File "tests/test_manager.py", line 60, in <module>
    import six
ModuleNotFoundError: No module named 'six'
status: 1
Traceback (most recent call last):
  File "tests/test_manager.py", line 60, in <module>
    import six
ModuleNotFoundError: No module named 'six'
status: 1
Traceback (most recent call last):
  File "tests/test_manager.py", line 60, in <module>
    import six
ModuleNotFoundError: No module named 'six'
status: 1
Thanks for creating a release. please check that the tests were sucessfull before releasing the version

I've uploaded it (temporarily — I will remove this section of this comment once I remove it later) to my CERN Box and it is public at this URL: https://feickert.web.cern.ch/feickert/tmp/MG5_aMC_v2.6.3.2.tar.gz

@kratsg Can you run a workflow with

curl -sLO https://feickert.web.cern.ch/feickert/tmp/MG5_aMC_v2.6.3.2.tar.gz

and let me know if the release I made is valid? If it is valid, then we should ask CERN IT to host it indefinitely.

kratsg commented 2 years ago

It was pointed out by a colleague in ATLAS that https://lcgpackages.web.cern.ch/tarFiles/sources/MCGeneratorsTarFiles/ exists as well.

matthewfeickert commented 2 years ago

It was pointed out by a colleague in ATLAS that https://lcgpackages.web.cern.ch/tarFiles/sources/MCGeneratorsTarFiles/ exists as well.

Though that sadly doesn't help as v2.6.3.2 isn't up there:

$ curl -s https://lcgpackages.web.cern.ch/tarFiles/sources/MCGeneratorsTarFiles/ | grep "MG5_aMC_v2.6.3.2.tar.gz" | wc -l
0
oliviermattelaer commented 2 years ago

The atlas repo does actually have evem more MG5 tarball that the amount that was on the uiuc server. I will try next week to transfer the v2.6.3.2 tarball from the uiuc server to the milan one (so the old link should be back working but only for a couple of month).

kratsg commented 2 years ago

Can't we migrate all of the old tarballs back onto launchpad? Surely they should stick around forever if you're going to use launchpad, or at least release them on github (you can add attachments to releases)

oliviermattelaer commented 2 years ago

I can discuss that point at our next mg developer meeting (in September). But for the moment, I will keep the strategy that only LTS version are staying forever on launchpad. plus the latest debug version of any feature release.

On 15 Jul 2022, at 21:30, Giordon Stark @.**@.>> wrote:

Can't we migrate all of the old tarballs back onto launchpad? Surely they should stick around forever if you're going to use launchpad, or at least release them on github (you can add attachments to releases)

— Reply to this email directly, view it on GitHubhttps://github.com/mg5amcnlo/mg5amcnlo/issues/16#issuecomment-1185844138, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH6535Q2QLEK2FQDGDORSVTVUG36RANCNFSM53ODG22Q. You are receiving this because you authored the thread.Message ID: @.***>

kratsg commented 2 years ago

I can discuss that point at our next mg developer meeting (in September). But for the moment, I will keep the strategy that only LTS version are staying forever on launchpad. plus the latest debug version of any feature release.

Fair, but this does mean that we end up publishing papers with very-hard-to-grab (old) MadGraph releases that are not necessarily LTS. I don't think any collaboration like ATLAS or CMS have a policy to only use LTS releases...

oliviermattelaer commented 2 years ago

uiuc server is currently down (also the ssh server) so I'm not able to transfer any of the old tar for the moment. Let's see if the server goes back online by itself (they might have a maintenance or something similar) otherwise I will have to ask my local contact to contact their IT team to see if they can do something.

oliviermattelaer commented 2 years ago

I got some information on the uiuc server. A power outage occured on friday and the system hosting our virtual machine/webise decided to fully re-install itself. In principle, they have backup of all data. But I guess that service will not be back online before next week at best.

oliviermattelaer commented 2 years ago

The tarball for 2.6.3.2 is now back accessible trough uiuc old web adress (via a redirection trough our new milan server)