Error reading time with CROCO files

[AndresSepulveda]

Hi,

I am testing the latest version of OpenDrift with a standard CROCO (v1.0) file and I get this error:

(opendrift) livecroco@livecroco:~/opendrift$ python3 example_croco_native.py 17:30:04 INFO opendrift: OpenDriftSimulation initialised (version 1.4.2) 17:30:04 INFO opendrift: Opening dataset: croco_his.nc 17:30:04 INFO opendrift: Opening file with Dataset 17:30:04 INFO opendrift: 'gls_cmu0' 17:30:04 INFO opendrift: Did not find complete set of GLS parameters Traceback (most recent call last): File "example_croco_native.py", line 15, in <module> nordic_native = reader_ROMS_native.Reader('croco_his.nc') File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in __init__ self.times = [datetime.utcfromtimestamp((OT - File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in <listcomp> self.times = [datetime.utcfromtimestamp((OT - numpy.core._exceptions.UFuncTypeError: ufunc 'subtract' cannot use operands with types dtype('float32') and dtype('<M8[s]')

Any clue what it is about?

Regards,

Andres

[knutfrode]

Hi, Can you try to update the conda environment with: conda env update -f environment.yml

On Tue, Nov 24, 2020, 21:32 Andres Sepulveda notifications@github.com wrote:

Hi,

I am testing the latest version of OpenDrift with a standard CROCO (v1.0) file and I get this error:

(opendrift) livecroco@livecroco:~/opendrift$ python3 example_croco_native.py 17:30:04 INFO opendrift: OpenDriftSimulation initialised (version 1.4.2) 17:30:04 INFO opendrift: Opening dataset: croco_his.nc 17:30:04 INFO opendrift: Opening file with Dataset 17:30:04 INFO opendrift: 'gls_cmu0' 17:30:04 INFO opendrift: Did not find complete set of GLS parameters Traceback (most recent call last): File "example_croco_native.py", line 15, in nordic_native = reader_ROMS_native.Reader('croco_his.nc') File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in init self.times = [datetime.utcfromtimestamp((OT - File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in self.times = [datetime.utcfromtimestamp((OT - numpy.core._exceptions.UFuncTypeError: ufunc 'subtract' cannot use operands with types dtype('float32') and dtype('<M8[s]')

Any clue what it is about?

Regards,

Andres

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/443, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABH25I3VVNXNUBXZM3WFTPTSRQJ5TANCNFSM4UBL773Q .

[AndresSepulveda]

Hi Knut,

I just did

`(opendrift) livecroco@livecroco:~/opendrift$ conda env update -f environment.yml Collecting package metadata (repodata.json): done Solving environment: done Installing pip dependencies: - Ran pip subprocess with arguments: ['/home/livecroco/miniconda3/envs/opendrift/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/livecroco/opendrift/condaenv.bgg3bnho.requirements.txt'] Pip subprocess output: Requirement already up-to-date: motuclient in /home/livecroco/miniconda3/envs/opendrift/lib/python3.8/site-packages (from -r /home/livecroco/opendrift/condaenv.bgg3bnho.requirements.txt (line 1)) (1.8.6)

done #

To activate this environment, use

#

$ conda activate opendrift

#

To deactivate an active environment, use

#

$ conda deactivate

` and then got again

(opendrift) livecroco@livecroco:~/opendrift$ python3 example_croco_native.py 17:52:25 INFO opendrift: OpenDriftSimulation initialised (version 1.4.2) 17:52:25 INFO opendrift: Opening dataset: croco_his.nc 17:52:25 INFO opendrift: Opening file with Dataset 17:52:31 INFO opendrift: 'gls_cmu0' 17:52:31 INFO opendrift: Did not find complete set of GLS parameters Traceback (most recent call last): File "example_croco_native.py", line 15, in nordic_native = reader_ROMS_native.Reader('croco_his.nc') File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in init self.times = [datetime.utcfromtimestamp((OT - File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 196, in self.times = [datetime.utcfromtimestamp((OT - numpy.core._exceptions.UFuncTypeError: ufunc 'subtract' cannot use operands with types dtype('float32') and dtype('<M8[s]')

[knutfrode]

Hm, a bit strange. Can you check with "git status" that all files are there? And you should have no pythonpath defined in combination with conda.

On Tue, Nov 24, 2020, 21:50 Andres Sepulveda notifications@github.com wrote:

Hi Knut,

I just did

`(opendrift) livecroco@livecroco:~/opendrift$ conda env update -f environment.yml Collecting package metadata (repodata.json): done Solving environment: done Installing pip dependencies: - Ran pip subprocess with arguments: ['/home/livecroco/miniconda3/envs/opendrift/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/livecroco/opendrift/condaenv.bgg3bnho.requirements.txt'] Pip subprocess output: Requirement already up-to-date: motuclient in /home/livecroco/miniconda3/envs/opendrift/lib/python3.8/site-packages (from -r /home/livecroco/opendrift/condaenv.bgg3bnho.requirements.txt (line 1)) (1.8.6)

done To activate this environment, use $ conda activate opendrift To deactivate an active environment, use $ conda deactivate

` and then got

`(opendrift) livecroco@livecroco:~/opendrift$ python3 example_croco_native.py Traceback (most recent call last): File "example_croco_native.py", line 8, in from opendrift.readers import reader_ROMS_native File "/home/livecroco/opendrift/opendrift/readers/reader_ROMS_native.py", line 23, in from basereader import BaseReader, vector_pairs_xy ModuleNotFoundError: No module named 'basereader'

`

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/443#issuecomment-733225862, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABH25I47QBM6DLGJ7VNBSA3SRQMA7ANCNFSM4UBL773Q .

[AndresSepulveda]

I corrected my previous post as I had tried to use an old readter_ROMS_native.py

Here is the ncdump log from my CROCO file ncdump_croco.log

Here is the requested git status

(opendrift) livecroco@livecroco:~/opendrift$ git status On branch master Your branch is up to date with 'origin/master'.

Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory)

modified:   opendrift/readers/reader_ROMS_native.py

Untracked files: (use "git add ..." to include in what will be committed)

croco_his.nc
example_croco_native.py
example_roms_native.py
ncdump_croco.log
opendrift/readers/reader_ROMS_native.py.orig

no changes added to commit (use "git add" and/or "git commit -a")

[AndresSepulveda]

Just to add, there is no PYTHONPATH defined in the system.

This was a fresh install of OpenDrift, barely a few hours old.

[knutfrode]

Can you also check whether all tests pass (testall or pytest)?

On Tue, Nov 24, 2020, 22:06 Andres Sepulveda notifications@github.com wrote:

Just to add, there is no PYTHONPATH defined in the system.

This was a fresh install of OpenDrift, barely a few hours old.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenDrift/opendrift/issues/443#issuecomment-733233188, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABH25I5IFJVVB3XSG5MZUT3SRQN6PANCNFSM4UBL773Q .

[AndresSepulveda]

Here it is

`opendrift) livecroco@livecroco:~/opendrift$ ./testall ======================================= test session starts ======================================= platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /home/livecroco/opendrift, configfile: pytest.ini, testpaths: tests plugins: cov-2.10.1, benchmark-3.2.3 collected 127 items

tests/integration/test_interpolation.py ........... [ 8%] tests/integration/test_leeway.py ... [ 11%] tests/integration/test_models.py ........ [ 17%] tests/integration/test_netcdf.py . [ 18%] tests/integration/test_oillibrary.py ....... [ 23%] tests/integration/test_physics.py ........ [ 29%] tests/integration/test_plot.py s [ 30%] tests/integration/test_readers.py ..................... [ 47%] tests/integration/test_run.py .................................... [ 75%] tests/integration/test_run_3d.py .. [ 77%] tests/integration/test_seed.py ... [ 79%] tests/integration/test_stranding.py .../testall: line 3: 4697 Killed pytest --ignore=tests/benchmarks --ignore=tests/wps --runslow --benchmark-disable $* (opendrift) livecroco@livecroco:~/opendrift$ `

[knutfrode]

The problem is that there is an error in the CROCO-files: the attribute time.units is simply seconds, whereas it should have the form seconds since 1948-01-01 00:00:00 (actual zero time should be inserted)

I had made a workaround in reader_ROMS_native for this earlier, but it did not apply when using Xarray, which is now default dependency.

But now I implemended this workaround also for the Xarray case (which is now also the only option in reader_ROMS_native). So if you do git pull once more, I expect it should work again.

Anyway, it would be better if CROCO-output was corrected, to avoid unhealthy hacks like this.

[AndresSepulveda]

Thank you again Knut, it works now.

CF compliance is a topic to address in the CROCO code, it is know, sorry.