Issue downloading domain data

[leifdenby]

below is from email from @jbrooke-metoffice

Hi Steef & Leif,

I’m finally resuming my SCM/MONC work with lagtraj but with a new dataset from a recent field campaign in Spain where we have a number of flux towers installed. I’ve tried to download a new domain, which includes land and sea but running into some problems. I was wondering if you have any suggestions, I remember running into a problem when I first downloaded the Cardington domain, and I think you mentioned the land/sea mask at the time.

liaise_julysop.yaml

source    : era5
lat_min   : 38.0
lat_max   : 43.5
lon_min   : -4.0
lon_max   : 4.0
lat_samp   : 0.1
lon_samp   : 0.1

(name_of_env) [jbrooke@vld034:/data/users/jbrooke/lagtraj-liaise]$ python -m lagtraj.domain.download lagtraj://liaise_julysop 2021/07/15 2021/07/17 --retry-rate 5

Running without numba

Status on current data requests:
/net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-15.nc:
                failed (97ef3714-cbcf-4fc7-99f7-def2833c4291)
/net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-16.nc:
                failed (d4601216-8d32-45ec-b5a9-b677101086e8)
/net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-17.nc:
                failed (775b14e2-dc03-4843-94c9-b2b90db21c6b)

I’m getting two errors:

(name_of_env) [jbrooke@vld034:/data/users/jbrooke/lagtraj-liaise]$ python -m lagtraj.domain.download lagtraj://liaise_julysop.yaml 2021/07/15 2021/07/16 --retry-rate 5

Running without numba
The requested domain (liaise_julysop.yaml) isn't currently available in lagtraj
(maybe you could add it with a pull-request at https://github.com/EUREC4A-UK/lagtraj/pulls)

The domains currently defined in lagtraj are:

And

UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-14: ordinal not in range(128)

Thanks,

Jenn

[leifdenby]

@jbrooke-metoffice thanks for the detailed notes on what you've tried!

I think what is going wrong is two things:

1) I recently noticed that I had made a mistake in setup.py which defines out lagtraj is installed with pip. This meant that the input yaml-files that we bundle with lagtraj weren't copied to the right place (that's the second error you're seeing). The good news is that I recently fixed this (https://github.com/EUREC4A-UK/lagtraj/pull/135), so if you reinstall lagtraj (i.e. uninstall and install) then this should be fixed:

python -m pip uninstall lagtraj
python -m pip install git+https://github.com/EUREC4A-UK/lagtraj

2) When you're calling lagtraj.domain.download you don't need the lagtraj:// prefix to the domain name. That prefix is only needed when you're using one of the files we've included with lagtraj. And instead of referring to the domain with the .yaml you just need the "name", which is the part before .yaml. The instructions should be clearer here sorry. I just created the liase_julysop.yaml file from your notes, the folder structure should look like this:

.
└── data
    └── domains
        ├── liaise_julysop_data
        │   └── data_requests.yaml
        └── liaise_julysop.yaml

And then you just call the download command with:

python -m lagtraj.domain.download liaise_julysop 2021/07/15 2021/07/17 --retry-rate 15

Let me know if this works :)

[mo-jbrooke]

Hi Leif,

Thanks for getting back to me so quickly. With the current version of lagtraj I already have downloaded your revised python download command seems to work, and I’ll get back to you if I run into any more problems.

On the installation side:

I had a few installation issues first time round when I did this, and I set up a conda environment with the various python packages that lagtraj needs, but are not available on the MO python software stack.

I haven’t uninstalled, but I tried to install in a different location and got:

(name_of_env) ***@***.***:/data/users/jbrooke]$ python -m pip install git+https://github.com/EUREC4A-UK/lagtraj
Collecting git+https://github.com/EUREC4A-UK/lagtraj
  Cloning https://github.com/EUREC4A-UK/lagtraj to /var/tmp/pip-req-build-eqwaxroo
  Running command git clone -q https://github.com/EUREC4A-UK/lagtraj /var/tmp/pip-req-build-eqwaxroo
    ERROR: Command errored out with exit status 1:
     command: /home/h03/jbrooke/miniconda3/envs/name_of_env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/var/tmp/pip-req-build-eqwaxroo/setup.py'"'"'; __file__='"'"'/var/tmp/pip-req-build-eqwaxroo/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /var/tmp/pip-pip-egg-info-mw5vp0fr
         cwd: /var/tmp/pip-req-build-eqwaxroo/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/var/tmp/pip-req-build-eqwaxroo/setup.py", line 20, in <module>
        long_description=open("README.md").read(),
      File "/home/h03/jbrooke/miniconda3/envs/name_of_env/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1259: ordinal not in range(128)
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I’ve also tried:

(name_of_env) ***@***.***:/scratch/jbrooke]$ conda install git+https://github.com/EUREC4A-UK/lagtraj
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - //github.com/eurec4a-uk/lagtraj

Installing unsupported python packages can be a bit tricky on this end. Are there any significant changes in lagtraj that I should persue the reinstall? I’m using a branch that Steef had made some changes for running at Cardington over land.

Cheers, Jenn

[leifdenby]

I've seen that error before, but I'm surprised it's happening and I haven't been able to reproduce it on my computer. Could you tell me what operating system and python version (i.e. the output of running python --version) you're using? I think this is a python2 related error.

It's quite an easy fix, but I just want to be able to reproduce it so I'm sure that it is fixed.

[leifdenby]

I haven't put lagtraj on conda (or pipy) yet, so conda install lagtraj or pip install lagtraj won't work :)

[mo-jbrooke]

I’m using Red Hat Enterprise Linux Client 7.9 and Python 3.6.10.

Unfortunately the ERA5 download has failed again.

Sleeping 15.0min... Retrying download

Status on current data requests: /net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-15.nc: failed (305345b1-6fb2-4dda-b24e-70a34343e93f) /net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-16.nc: failed (12d7fdc3-c323-4066-a6e7-adcc7e4319e7) /net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-17.nc: failed (f4d41d17-fffd-4c93-a7df-dcdfbb839ffc)

The data_requests.yaml is as follows:

/net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-15.nc: query_hash: '13139170' request_id: 305345b1-6fb2-4dda-b24e-70a34343e93f /net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-16.nc: query_hash: '13532397' request_id: 12d7fdc3-c323-4066-a6e7-adcc7e4319e7 /net/data/users/jbrooke/lagtraj-liaise/data/domains/liaise_julysop_data/an_model_2021-07-17.nc: query_hash: '13401324' request_id: f4d41d17-fffd-4c93-a7df-dcdfbb839ffc

Cheers, Jenn

From: Leif Denby @.> Sent: 17 August 2021 12:42 To: EUREC4A-UK/lagtraj @.> Cc: Brooke, Jennifer @.>; Mention @.> Subject: Re: [EUREC4A-UK/lagtraj] Issue downloading domain data (#138)

This email was received from an external source. Always check sender details, links & attachments.

I've seen that error before, but I'm surprised it's happening and I haven't been able to reproduce it on my computer. Could you tell me what operating system and python version (i.e. the output of running python --version) you're using? I think this is a python2 related error.

It's quite an easy fix, but I just want to be able to reproduce it so I'm sure that it is fixed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/EUREC4A-UK/lagtraj/issues/138#issuecomment-900223581, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARXI44P754RPEWYXF6DUVXDT5JDG5ANCNFSM5CJPRWVA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[leifdenby]

@jbrooke-metoffice I'm going to close this assuming this is no longer an issue. Please re-open if not :)