"transform_worker" could not be found (but it's there!)

[francescaLoi]

I am getting this error while running a simple .yml file:

Traceback (most recent call last):
  File "/home/floi/cara_master/caracal/caracal/workers/worker_administrator.py", line 115, in __init__
    wkr = __import__(_worker)
  File "/home/floi/cara_master/caracal/caracal/workers/transform_worker.py", line 12, in <module>
    from caracal.workers.utils.callibs import resolve_calibration_library
  File "/home/floi/cara_master/caracal/caracal/workers/utils/callibs.py", line 4, in <module>
    from collections import OrderedDict, Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
2023-02-01 10:10:23 CARACal ERROR: Worker "transform_worker" could not be found at /home/floi/cara_master/caracal/caracal/workers [ImportError]
2023-02-01 10:10:23 CARACal INFO:   More information can be found in the logfile at log-caracal.txt
2023-02-01 10:10:23 CARACal INFO: exiting with error code 1

It never happened before, I am using CARACal really often and the error appeared today with the same installation that was working until yesterday. I also tried with a fresh installation, stable version, github master version, same error. I can understand it looking at the file /usr/lib/python3.10/collections/__init__.py since the Iterable function required by the callibs.py file is missing but as far as I can see, these file haven't changed recently. Any idea?

[Athanaseus]

Hi @francescaLoi

I just looked this up; it has been moved from Collections Abstract Base Classes to the collections.abc module in py3.10, and you can use Iterable from collections.abc instead. Note that the pipeline hasn't been tested for py3.10 compatibility yet.

For backward compatibility, these classes remain visible in this module through Python 3.9.

Cheers

[francescaLoi]

Thank you @Athanaseus ! Sorry, I didn't know. With python3.9 it works as usual.

[PeterKamphuis]

I just ran into this issue with python 3.11. If I understand this correctly this means that we now have a caracal master that only runs on python3.8 and python3.9. That really is super restrictive in versions. I don't think we should want this. In anycase we should at least make that clear to people then.

[PeterKamphuis]

@Athanaseus it seems that the abc module has been part of collections since python 3.3 should in principle this should work if these always get imported from collections.abc right?

[Athanaseus]

@PeterKamphuis, thank you for bringing my attention to this issue.

From this error,

File "/home/floi/cara_master/caracal/caracal/workers/utils/callibs.py", line 4, in <module>
    from collections import OrderedDict, Iterable

it means the import needs to be the following

try:
    from collections.abc import Iterable # py >= 3.10
except ImportError:
    from collections import Iterable # py < 3.10

Check the discussion here: cannot-import-name-iterable

I will consider this as I prepare for the next caracal release aimed to support 3.7-3.10, and possibly 3.11. Just so you know, 3.7 is still supported. I will also update the docs for this.

PS: Python3.12 just got released yesterday.

[PeterKamphuis]

Hi @Athanaseus,

I don't think 3.7 is still supported. At least I got a terrible error trying to install caracal? However I'm still getting error so it might be a problem on my side.

I started a branch to make this work on python 3.11 (run_with3_11). As far as I could see from the documentation [https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes]() abc was introduced in python 3.3 so it seemed to me that simply importing it from there is better than a try statement?

[Athanaseus]

@PeterKamphuis you are right; indeed, it was introduced since 3.3. The collections module was removed in 3.10 after being deprecated in 3.3. So yes, importing collections.abc instead should solve the issue.

According to the testing workflows, 3.7 installations are going through, maybe you need to run with the current master to test this.

[PeterKamphuis]

@Athanaseus I tried installing with 3.7 and it installs ok but I can't test at the moment as my stimela pull -s -f is not going through (See https://github.com/ratt-ru/Stimela/issues/620). But that is the same for 3.11.

Fact remains though that we state that caracal is 3.8+ and it is not.

[Athanaseus]

resolved in #1547