easy_update not updating anything

[casparvl]

I'm trying to use easy update (v 2.1.3), but no .update file is being produced for me.

~/TMP/easy_update-2.1.3/easy_update.py --verbose /sw/noarch/Debian10/2020/software/EasyBuild/4.4.0/easybuild/easyconfigs/j/JupyterHub/JupyterHub-1.1.0-GCCcore-10.2.0.eb
reading dependency: Python-3.8.6-GCCcore-10.2.0.eb
$

I would expect this to create an JupyterHub-XYZ.update easyconfig in the current directory, and also I'd expect it to print a lot more verbose output.

What am I missing here? Am I not using it correctly? It should be usable on PythonBundle's right?

[fizwit]

Can you send me gist of the file you want to update? I will test

On Wed, Jul 7, 2021 at 5:53 AM Caspar van Leeuwen @.***> wrote:

I'm trying to use easy update (v 2.1.3), but no .update file is being produced for me.

~/TMP/easy_update-2.1.3/easy_update.py --verbose /sw/noarch/Debian10/2020/software/EasyBuild/4.4.0/easybuild/easyconfigs/j/JupyterHub/JupyterHub-1.1.0-GCCcore-10.2.0.eb reading dependency: Python-3.8.6-GCCcore-10.2.0.eb $

I would expect this to create an JupyterHub-XYZ.update easyconfig in the current directory, and also I'd expect it to print a lot more verbose output.

What am I missing here? Am I not using it correctly?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fizwit/easy_update/issues/33, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHC3SYFNXKGKGJN5L5EQOTTWRE4BANCNFSM476TM3AA .

-- John Dey @.***

[casparvl]

It's this official EasyConfig: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/j/JupyterHub/JupyterHub-1.1.0-GCCcore-10.2.0.eb

but I also tried e.g. https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/i/IPython/IPython-7.25.0-GCCcore-10.3.0.eb with pretty much the same result (i.e. it prints reading dependency: Python-3.9.5-GCCcore-10.3.0.eb and then just returns). So, it doesn't really seem to be the specific file.

I read in your November 2020 update notes 'The primary search path will based on the path of the earyconfig being updated.'. I'm not entirely sure what you mean by that, does it mean that the EasyConfig that I try to update should be in the main directory of EasyConfigs (i.e. the ones installed by EasyBuild)? Or can I just have some EasyConfig in a completely seperate dir (say /tmp for the sake of argument) and run it on that?

Also: does the script make any assumptions on the directory in which it is run? I suppose only that it's writeable, right - since I suppose the current dir is where it will output the .update file?

[casparvl]

small update: I just changed the logging level to logging.DEBUG in the framework.py since I saw that you write quite some things to logging.debug(...). It gave me this on the IPython EasyConfig:

$ ~/TMP/easy_update-2.1.3/easy_update.py --verbose /sw/noarch/Debian10/2021/software/EasyBuild/4.4.1/easybuild/easyconfigs/i/IPython/IPython-7.25.0-GCCcore-10
.3.0.eb
DEBUG:root:EB path: /sw/noarch/Debian10/2021/software/EasyBuild/4.4.1/bin/eb
DEBUG:root:EB search path: /sw/noarch/Debian10/2021/software/EasyBuild/4.4.1/easybuild/easyconfigs:/sw/noarch/Debian10/2021/software/EasyBuild/4.4.1/easybuild/easyconfigs
DEBUG:root:primary language Python 3.9.5
DEBUG:root:Python Toolchains: ['GCCcore-10.3.0']
DEBUG:root:build_dep_filename ['Python-3.9.5-GCCcore-10.3.0.eb']
DEBUG:root:language file: ['Python-3.9.5-GCCcore-10.3.0.eb']
DEBUG:root:find_easyconfig: search for Python-3.9.5-GCCcore-10.3.0.eb in /sw/noarch/Debian10/2021/software/EasyBuild/4.4.1/easybuild/easyconfigs/p/Python
reading dependency: Python-3.9.5-GCCcore-10.3.0.eb
DEBUG:root:interperting EasyConfig error: name 'SOURCE_WHL' is not defined

I check Python-3.9.5-GCCcore-10.3.0.eb, and indeed it contains e.g. an extension using this SOURCE_WHL template:

('pastel', '0.2.1', {
        'source_tmpl': SOURCE_WHL,
        'checksums': ['4349225fcdf6c2bb34d483e523475de5bb04a5c10ef711263452cb37d7dd4364'],
    }),

I was guessing the fact that that isn't yet in your list of template items added to header in parse_eb (in framework.py) must be the problem right? (see https://docs.easybuild.io/en/latest/version-specific/easyconfig_templates.html for the meaning of SOURCE_WHL)

Anyway, I tried adding the lines:

        header += 'SOURCE_WHL = "%(name)s-%(version)s-py2.py3-none-any.whl"\n'
        header += 'SOURCE_PY3_WHL = "%(name)s-%(version)s-py3-none-any.whl"\n'

and now it seems to complete succesfully. (well... next... I got a write permission error because it tries to write in my EB install prefix, where I don't have write permissions)

If I might make two suggestions I would

1. Add a --debug command line option that sets the logging level to debug
2. Add an option to write the *.update file in a different directory than the original EB (or change the default output dir to the current directory). Seems to me like the original EB might be in a non-writeable location (i.e. a system installation) in many cases.

If you want I can try to make a PR for the two lines above, but it might be faster if you just push it yourself. Let me know what you prefer.

[casparvl]

Btw, I don't wanna give off the wrong signal here with my comments & suggestions: I love the tool! This will save me a lot of messing-around by hand...! :)

[fizwit]

Thanks for working on this. As you can see most of easy_update is code is performing tasks that are in EasyBuild. I need to migrate the update into EB this fall. I have been planning this for years, but I really need to make this happen.

• Templates are an issue, I add them by hand as easy_update breaks. I don't have all of them.

  • I used to have a debug flag I will add that back into the args

  • I do not want to add an --output flag since there is no similar function in EB. I assume the users are working inside a clone of the EasyConfig, that they are updating easyconfig and have priv to write/change files. (similar to --inject-checksum).

  • I will check for EBROOTEASYBUID in the environment. Then you can run from outside the repo

Python Issues, Since 2019B the version suffix have been removed. I used the EasyConfig file name to check for additional dependencies. Without Python version suffixes every file in dependencies needs to be checked.

John Dey @.***

On Wed, Jul 7, 2021 at 5:53 AM Caspar van Leeuwen @.***> wrote:

I'm trying to use easy update (v 2.1.3), but no .update file is being produced for me.

~/TMP/easy_update-2.1.3/easy_update.py --verbose /sw/noarch/Debian10/2020/software/EasyBuild/4.4.0/easybuild/easyconfigs/j/JupyterHub/JupyterHub-1.1.0-GCCcore-10.2.0.eb reading dependency: Python-3.8.6-GCCcore-10.2.0.eb $

I would expect this to create an JupyterHub-XYZ.update easyconfig in the current directory, and also I'd expect it to print a lot more verbose output.

What am I missing here? Am I not using it correctly?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fizwit/easy_update/issues/33, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHC3SYFNXKGKGJN5L5EQOTTWRE4BANCNFSM476TM3AA .

[fizwit]

There was a bug in find_easyconfigs_paths. You can work outside of the easyconfigs path, which will allow you write you files.

--debug has been added

use tag 2.1.5

thanks for the help

John Dey @.***

On Thu, Jul 8, 2021 at 11:46 AM John Dey @.***> wrote:

Thanks for working on this. As you can see most of easy_update is code is performing tasks that are in EasyBuild. I need to migrate the update into EB this fall. I have been planning this for years, but I really need to make this happen.

• Templates are an issue, I add them by hand as easy_update breaks. I don't have all of them.

  • I used to have a debug flag I will add that back into the args

  • I do not want to add an --output flag since there is no similar function in EB. I assume the users are working inside a clone of the EasyConfig, that they are updating easyconfig and have priv to write/change files. (similar to --inject-checksum).

  • I will check for EBROOTEASYBUID in the environment. Then you can run from outside the repo

Python Issues, Since 2019B the version suffix have been removed. I used the EasyConfig file name to check for additional dependencies. Without Python version suffixes every file in dependencies needs to be checked.

John Dey @.***

On Wed, Jul 7, 2021 at 5:53 AM Caspar van Leeuwen < @.***> wrote:

I'm trying to use easy update (v 2.1.3), but no .update file is being produced for me.

~/TMP/easy_update-2.1.3/easy_update.py --verbose /sw/noarch/Debian10/2020/software/EasyBuild/4.4.0/easybuild/easyconfigs/j/JupyterHub/JupyterHub-1.1.0-GCCcore-10.2.0.eb reading dependency: Python-3.8.6-GCCcore-10.2.0.eb $

I would expect this to create an JupyterHub-XYZ.update easyconfig in the current directory, and also I'd expect it to print a lot more verbose output.

What am I missing here? Am I not using it correctly?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fizwit/easy_update/issues/33, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHC3SYFNXKGKGJN5L5EQOTTWRE4BANCNFSM476TM3AA .

[casparvl]

As you can see most of easy_update is code is performing tasks that are in EasyBuild. I need to migrate the update into EB this fall. I have been planning this for years, but I really need to make this happen.

Yeah I can imagine it's difficult to find time for something like this, but I do think it would be really great if it is integrated in EB. It would reach more people, but most importantly: it would indeed save you from having to mirror the behaviour of EB in terms of templates etc.

Ow well, having the current script at least already saves me a ton of time in manually updating versions :)