Limit NumPy version < 2

zxdawn commented 1 month ago

I have tested this notebook with a new conda environment and got an error for this cell:

%%sh
qd2hp -outdir . -fitwin hcho_analysis -slcol "ch2o=HCHO" S5P_L2_QDOASSCD_radasref.nc

Traceback (most recent call last):
  File "[/home/xinz/miniconda3/envs/avl/bin/qd2hp", line 10](http://localhost:2024/home/xinz/miniconda3/envs/avl/bin/qd2hp#line=9), in <module>
    sys.exit(cml())
             ^^^^^
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/qdoas2harp/convert2harp.py", line 43](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/qdoas2harp/convert2harp.py#line=42), in cml
    qdoas_obj.print_product(outdir)
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/qdoas2harp/convert2harp.py", line 120](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/qdoas2harp/convert2harp.py#line=119), in print_product
    harp.export_product(self.product,outputfile , file_format='hdf5', hdf5_compression=6)
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py", line 1399](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py#line=1398), in export_product
    _export_product(product, c_product_ptr[0])
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py", line 1111](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py#line=1110), in _export_product
    _export_variable(name, product[name], c_product)
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py", line 1038](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py#line=1037), in _export_variable
    c_data_type_valid_min = _get_c_data_type(valid_min)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py", line 647](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/harp/_harppy.py#line=646), in _get_c_data_type
    elif numpy.can_cast(value, numpy.int8):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: can_cast() does not support Python ints, floats, and complex because the result used to depend on the value.
This change was part of adopting NEP 50, we may explicitly allow them again in the future.
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
Cell In[6], line 1
----> 1 get_ipython().run_cell_magic('sh', '', 'qd2hp -outdir . -fitwin hcho_analysis -slcol "ch2o=HCHO" S5P_L2_QDOASSCD_radasref.nc\n')

File [~/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/interactiveshell.py:2541](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/interactiveshell.py#line=2540), in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2539 with self.builtin_trap:
   2540     args = (magic_arg_s, cell)
-> 2541     result = fn(*args, **kwargs)
   2543 # The code below prevents the output from being displayed
   2544 # when using magics with decorator @output_can_be_silenced
   2545 # when the last Python token in the expression is a ';'.
   2546 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File [~/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/magics/script.py:155](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/magics/script.py#line=154), in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
    153 else:
    154     line = script
--> 155 return self.shebang(line, cell)

File [~/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/magics/script.py:315](http://localhost:2024/home/xinz/miniconda3/envs/avl/lib/python3.12/site-packages/IPython/core/magics/script.py#line=314), in ScriptMagics.shebang(self, line, cell)
    310 if args.raise_error and p.returncode != 0:
    311     # If we get here and p.returncode is still None, we must have
    312     # killed it but not yet seen its return code. We don't wait for it,
    313     # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
    314     rc = p.returncode or -9
--> 315     raise CalledProcessError(rc, cell)

CalledProcessError: Command 'b'qd2hp -outdir . -fitwin hcho_analysis -slcol "ch2o=HCHO" S5P_L2_QDOASSCD_radasref.nc\n'' returned non-zero exit status 1.

It's related to the new numpy version. The old numpy version 1.26.4 works well.

tdanckaert commented 1 month ago

Hi, thanks for reporting this issue. We'll have a look at how to best improve this (ideally, we adjust the code so it works with the latest numpy, too).

svniemeijer commented 1 month ago

First, the error is raised by qd2hp, so the issue would belong in https://github.com/UVVIS-BIRA-IASB/qdoas2harp and not here.

However, the traceback is in HARP, so I think it has to be solved there. If a numpy version limitation needs to be applied this then likely needs to be applied within the HARP package. But more generally, everything that uses numpy should be checked on compatibility with numpy v2

tdanckaert commented 17 hours ago

Hi, I believe this issue is fixed in harp commit https://github.com/stcorp/harp/commit/910cb95d6c02f84e64f01f7b868e26bf915b7c73 . Running with harp >= 1.30 should resolve the problem.

UVVIS-BIRA-IASB / qdoas

Limit NumPy version < 2 #3