RuntimeError: OrderedDict mutated during iteration during desk sed

[TomGoffrey]

Describe the bug

desk sed produces RuntimeError: OrderedDict mutated during iteration. This command works fine on my OS X laptop, but fails (as shown below) on my Ubuntu desktop.

To Reproduce

$ desk fit --source='desk/put_target_data_here/MSX_LMC_807.csv' --grid='oxygen' --n=50 --distance=10

You already have the grid!

Scaling to full grid (100,000 models)

             Target: MSX_LMC_807        1/1
--------------------------------------------------------
Luminosity          |   1,000 Msun
Optical depth (at 10 um)    |   3.16
Expansion velocity (scaled) |   5.96 km/s
Gas mass loss (scaled)      |   3.41E-06 Msun/yr
--------------------------------------------------------
tom@tom-pc:~/dev/Dusty-Evolved-Star-Kit$ desk sed
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|===========================================================================================================================================================| 1.3k/1.3k (100.00%)         0s
Traceback (most recent call last):
  File "/home/tom/.local/bin/desk", line 11, in <module>
    sys.exit(main())
  File "/home/tom/dev/Dusty-Evolved-Star-Kit/desk/main.py", line 33, in main
    func(**funcargs)
  File "/home/tom/dev/Dusty-Evolved-Star-Kit/desk/console_commands.py", line 63, in sed
    plotting_seds.create_fig(source_path, source_filename, dest_path, save_name)
  File "/home/tom/dev/Dusty-Evolved-Star-Kit/desk/outputs/plotting_seds.py", line 197, in create_fig
    input_file = Table.read(source_path + "/" + source_filename)
  File "/home/tom/.local/lib/python3.6/site-packages/astropy/table/connect.py", line 52, in __call__
    out = registry.read(cls, *args, **kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/astropy/io/registry.py", line 520, in read
    'read', cls, path, fileobj, args, kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/astropy/io/registry.py", line 595, in _get_valid_format
    valid_formats = identify_format(mode, cls, path, fileobj, args, kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/astropy/io/registry.py", line 414, in identify_format
    for data_format, data_class in _identifiers:
RuntimeError: OrderedDict mutated during iteration

If you suspect this is an IPython 7.16.1 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Expected behavior Command produces plot saved in file.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

• Part of openjournals/joss-reviews#2554
• Current master, dc6012c.
• Python 3.6.9, Ubuntu 18.04.05.
• Connected to desktop over ssh. I doubt it's relevant, but I'll mention it anyway.

[TomGoffrey]

FWIW the laptop that this does run on is running python 3.7.6, so I'm wondering if there's a 3.6 vs 3.7 change in behaviour?

[s-goldman]

@TomGoffrey Hmmm, what version of astropy are you using? The tests check a Ubuntu system running python 3.6. Does this repeat every time you run it on this setup?

[TomGoffrey]

astropy version is '4.0.1.post1'

The error repeats every time I try from command line.

Some slight weirdness from trying to trigger the issue in ipython3 (by running import desk;desk.sed()), the first time fails in the same manner as before, but the second time I run it appears to work fine. This pattern seems repeatable too.

I'm leaning towards something being not quite right with the machine I'm testing on. I'll have physical access to it tomorrow, so I'll try and investigate more then. If anything I can at least restart it for the first time in 250 days, lockdown hasn't been kind to it.

[TomGoffrey]

In case it's useful, here's the output from running the commands in ipython3. There's some complaints at the end about running sys.exit() but the .png file is clearly new.

In [1]: import desk

In [2]: desk.sed()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-7fde54a9a56d> in <module>
----> 1 desk.sed()

~/dev/Dusty-Evolved-Star-Kit/desk/console_commands.py in sed(source_path, source_filename, dest_path, save_name)
     61         SED figure with data in blue and model in black.
     62     """
---> 63     plotting_seds.create_fig(source_path, source_filename, dest_path, save_name)
     64
     65

~/dev/Dusty-Evolved-Star-Kit/desk/outputs/plotting_seds.py in create_fig(source_path, source_filename, dest_path, save_name)
    195     """
    196
--> 197     input_file = Table.read(source_path + "/" + source_filename)
    198     n = len(input_file)  # number of fit sources
    199

~/.local/lib/python3.6/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     50     def __call__(self, *args, **kwargs):
     51         cls = self._cls
---> 52         out = registry.read(cls, *args, **kwargs)
     53
     54         # For some readers (e.g., ascii.ecsv), the returned `out` class is not

~/.local/lib/python3.6/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs)
    518
    519             format = _get_valid_format(
--> 520                 'read', cls, path, fileobj, args, kwargs)
    521
    522         reader = get_reader(format, cls)

~/.local/lib/python3.6/site-packages/astropy/io/registry.py in _get_valid_format(mode, cls, path, fileobj, args, kwargs)
    593     """
    594
--> 595     valid_formats = identify_format(mode, cls, path, fileobj, args, kwargs)
    596
    597     if len(valid_formats) == 0:

~/.local/lib/python3.6/site-packages/astropy/io/registry.py in identify_format(origin, data_class_required, path, fileobj, args, kwargs)
    412     """
    413     valid_formats = []
--> 414     for data_format, data_class in _identifiers:
    415         if _is_best_match(data_class_required, data_class, _identifiers):
    416             if _identifiers[(data_format, data_class)](

RuntimeError: OrderedDict mutated during iteration

In [3]: desk.sed()
An exception has occurred, use %tb to see the full traceback.

SystemExit

/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3351: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

In [4]: ls -l *.png
-rw------- 1 tom tom 56750 Sep 29 10:40 output_sed_MSX_LMC_807.png

In [5]: from datetime import datetime;datetime.now().strftime("%d/%m/%Y %H:%M:%S")
Out[5]: '29/09/2020 10:40:48'

[s-goldman]

Well, I've figured out the second system exit issue, but not the initial mutated Ordered dictionary issue. The code was working, but the output text for the python interface was more jarring than was intended. Thanks for finding this and pointing it out. I've fixed it in #172.

[TomGoffrey]

Thanks, the second call to desk.sed() runs cleanly for me now. I'll update tomorrow if/when I make progress on the original issue.

[s-goldman]

Great, thanks Tom.

[TomGoffrey]

OK, after a solid 6 hours of debugging and arguing with Docker, I've tracked down the issue on the problematic machine to a rogue installation of an old version of asdf living in /usr/lib/python3/dist-packages.

Sorry for the false alarm.

[s-goldman]

@TomGoffrey Thanks for putting in the work to track it down!