Error running US-ARc example files

[pisaac-ozflux]

I get the following error when running the latest ONEFlux with the US-ARc example files: ONEFluxError: US-ARc mismatched last ERA timestamp expected (202112312330) and found (201412312330)

Details: ONEFlux: Cloned from GitHub on 27th October 2022 US-ARc: ftp files downloaded on 26th October 2022 OS: Ubuntu 20.04 Python: conda environment using V2.7.15

I have saved the log file and can attach this if it will help.

Anyone else have this problem?

[gilbertozp]

Hi Peter,

This is because of a change in the default configurations for the default first and last years for the ERA product; right now this is being done through the oneflux/local_settings.py file, which you can add to your local clone, or via a command line parameter that overrides the settings and hardcoded defaults, e.g. to run the exemple US-ARc files, you'd use: runoneflux.py [...] --era-fy 1989 and --era-ly 2014

We're reworking the parameterization of runs, but in the meantime, I'll plan to fix the documentation to reflect this.

Thanks for raising the issue.

[gilbertozp]

@gilbertozp: add information about parameters and local_settings.py file to main documentation

[pisaac-ozflux]

Hi Gilberto,

Many thanks for the reply and the work-around. I will try it out now.

You must be working late. According to me, it's 2300 in Berkeley!

Cheers, Peter

On Fri, 28 Oct 2022 at 16:48, Gilberto Pastorello @.***> wrote:

Hi Peter,

This is because of a change in the default configurations for the default first and last years for the ERA product; right now this is being done through the oneflux/local_settings.py file, which you can add to your local clone, or via a command line parameter that overrides the settings and hardcoded defaults, e.g. to run the exemple US-ARc files, you'd use: runoneflux.py [...] --era-fy 1989 and --era-ly 2014

We're reworking the parameterization of runs, but in the meantime, I'll plan to fix the documentation to reflect this.

Thanks for raising the issue.

— Reply to this email directly, view it on GitHub https://github.com/fluxnet/ONEFlux/issues/39#issuecomment-1294491830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTSJCIWRSPOFOOU2RO3IDDWFNSLHANCNFSM6AAAAAARPRBL7M . You are receiving this because you authored the thread.Message ID: @.***>

-- Peter Isaac Ph: +61 3 59685998 Mob: 0429053970 Skype: pisaac.ozflux Email: @.***

"Vita brevis, ars longa, occasio praeceps, experimentum periculosum, iudicium difficile" - Hippocrates 400BC "What's the go o'that?" - James Clerk Maxwell c1835

[pisaac-ozflux]

Hey Gilberto,

Using --era-fy and era-ly fixed the problem and I was able to get recent data from an OzFlux site through the ONEFlux pipeline for the first time. Very satisfying!

I struck a second problem in the process, see the traceback below: File "runoneflux.py", line 94, in era_first_year=args.erafy, era_last_year=args.eraly) File "/home/peter/ONEFlux/oneflux/tools/pipeline.py", line 85, in run_pipeline pipeline.run() File "/home/peter/ONEFlux/oneflux/pipeline/wrappers.py", line 262, in run driver.run() File "/home/peter/ONEFlux/oneflux/pipeline/wrappers.py", line 2009, in run self.convert_files() File "/home/peter/ONEFlux/oneflux/pipeline/wrappers.py", line 1970, in convert_files data = numpy.genfromtxt(fname=nt_filename, names=True, delimiter=',', skip_header=0, usemask=False, usecols=(reco_idx, gpp_idx,)) File "/home/peter/miniconda3/envs/runoneflux/lib/python2.7/site-packages/numpy/lib/npyio.py", line 1963, in genfromtxt for (i, line) in enumerate(itertools.chain([first_line, ], fhd)): File "/home/peter/miniconda3/envs/runoneflux/lib/python2.7/codecs.py", line 314, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc7 in position 146: invalid continuation byte

The fix was to specify the same encoding ("latin-1") in the writing and reading of the file as below:

1. in the call to numpy.savetxt() in nighttime.partitioning_nt(), see below; numpy.savetxt(fname=output_filename, X=result_year_data, delimiter=',', fmt='%s', header=','.join(result_year_data.dtype.names), comments='', encoding="latin-1")

2. in the call to numpy.genfromtxt() in wrappers.PipelinePrepareURE.convert_files(), see below; data = numpy.genfromtxt(fname=nt_filename, names=True, delimiter=',', skip_header=0, usemask=False, usecols=(reco_idx, gpp_idx,), encoding="latin-1")

It seems that numpy.savetxt() uses "latin-1" by default but numpy.genfromtxt() uses "bytes" by default. The docs say this should work but maybe something is falling through a crack. Everything seems to run OK with the above change.

Do you want me to log this as a separate issue?

And by the way, many, many thanks for your efforts on ONEFlux over the years. The longer I spend wandering around inside the code the more respect I have for what you've done.

Cheers, Peter

[q996261913]

Peter, could you ask how you run data from other years? I have my own half hour flux data for 2022-2023, but the ERA data provided in Part 06 of the sample data is from 1984 to 2014. What can I do to run data from 2022-2023?