'Metadata' not found in sequence metadata.txt

[saberioon]

Any advice on how to fix this error?

`Processing sequence: /SEQ20210201T132732 Reading raw data... seq /SEQ20210201T132732/metadata.txt Failed: KeyError('Metadata') Traceback (most recent call last): File "/hypernets_processor/hypernets_processor/main/sequence_processor_main.py", line 126, in main sp.process_sequence(target_sequence) File "hypernets_processor/hypernets_processor/sequence_processor.py", line 105, in process_sequence l0_irr,l0_rad,l0_bla,l0_swir_irr,l0_swir_rad,l0_swir_bla = reader.read_sequence(sequence_path,calibration_data_rad,calibration_data_irr,calibration_data_swir_rad,calibration_data_swir_irr) File "/Users/mms/Projects/hypernets_processor/hypernets_processor/data_io/hypernets_reader.py", line 896, in read_sequence seq,lat,lon,cc,metadata,seriesIrr,seriesRad,seriesBlack,seriesPict,flag = self.read_metadata( File "/Users/mms/Projects/hypernets_processor/hypernets_processor/data_io/hypernets_reader.py", line 818, in read_metadata globalattr = dict(metadata['Metadata']) File "/Users/mms/opt/miniconda3/envs/hyProcessor/lib/python3.8/configparser.py", line 960, in getitem raise KeyError(key) KeyError: 'Metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/mms/opt/miniconda3/envs/hyProcessor/bin/hypernets_sequence_processor", line 33, in sys.exit(load_entry_point('hypernets-processor', 'console_scripts', 'hypernets_sequence_processor')()) File "/Users/mms/Projects/hypernets_processor/hypernets_processor/cli/sequence_processor_cli.py", line 122, in cli main(processor_config_path=PROCESSOR_CONFIG_PATH, job_config_path=job_config_path, to_archive=False) File "/Users/mms/Projects/hypernets_processor/hypernets_processor/main/sequence_processor_main.py", line 132, in main context.anomaly_db.add_x_anomaly() AttributeError: 'NoneType' object has no attribute 'add_x_anomaly'`

[jkuusk]

Sam posted this solution to GLPI (ticket no. 75) --8<-- This is caused because the current version of hypernets_tools creates an incomplete metadata.txt file. It should work if you add to the top of SEQXXXXXX/metadata.txt something like:

[Metadata]
PyxisVersion=PYXIS_V000.50
Datetime=20201117T144353
PI=CPE2
Site_name=Villefranche-sur-mer
Lat=43.69862
Lon=7.30692

--8<--

[saberioon]

Sam posted this solution to GLPI (ticket no. 75) --8<-- This is caused because the current version of hypernets_tools creates an incomplete metadata.txt file. It should work if you add to the top of SEQXXXXXX/metadata.txt something like:

[Metadata]
PyxisVersion=PYXIS_V000.50
Datetime=20201117T144353
PI=CPE2
Site_name=Villefranche-sur-mer
Lat=43.69862
Lon=7.30692

--8<--

Thanks for the solution. Unfortunately, my access to GLPI is very limited, and cannot see other tickets except mine. can you tell me what is PI ? and should I keep PyxisVersion=PYXIS_V000.50 as it is?

[jkuusk]

I'm not sure whether the metadata values alter the behaviour of the processor (except in the case they are completely missing). PI is principal investigator, I set it to my initials PI=JK and left the PyxisVersion as Sam suggested, although it probably won't make any difference as the Pyxis code is not used any more in the current hypernets_tools version.

[shunt16]

Thanks Joel, yes that's right. As far as I'm aware the PyxisVersion value doesn't affect the processing, I just took that template from a previously working file.

[shunt16]

@corizzi (cc @clemgoyens) is there a plan to reinstate the metadata to the metadata.txt file? Or do we need to change our processing somehow?

[clemgoyens]

Hi all, yes we discussed this issue yesterday with Alex. It is a request we made a long time ago but unfortunately, due to misunderstanding, it wasn't clear that it was an urgent request. This issue should be solved soon!

Or do we need to change our processing somehow? Well I think this is less robust. E.g. we read the SN of the HYPSTAR from this header to look for the appropriate calibration file. Also datetime and latitude and longitude should come from the GPS of the system. What do you think?

[clemgoyens]

@saberioon To process currently I do the following to add metadata headers to all my sequences (dirty solution but working for now!): run ./add_header.sh

_with addheader.sh being:

!/bin/bash

for file in $(find DATA/DATA/*/metadata.txt); do echo Processing $file

cat blankaart_header.txt $file > $file.modified

mv $file.modified $file

done

_and blankaartheader.txt (only lat and lon is used, other fields are currently used to fill in the metadata of the netcdf): [Metadata] PyxisVersion=PYXIS_V000.50 Datetime=NAN PI=Clem Site_name=Blankaart_South Lat=50.98877462132464 Lon=2.8358506152766276

[shunt16]

Great thanks @clemgoyens and @corizzi

On the most robust place for this metadata to be read from:

• PyxisVersion - don't know what this is or what it's for
• Datetime, Lat, Lon - I agree for the location and time data should be read from the GPS, I don't think this is transmitted yet though?
• Serial Number - Not sure if there's a better solution for serial number. Perhaps Alex could somehow read it from the instrument for the most robust solution.
• PI and Site_name - I think should come from the processor config file since PI and site_name are only concepts relevant within the context of the network

So I think for sure we need the serial number, and the date time and location for now until we get the GPS data.

[jkuusk]

Timestamp is in the header of each spectrum.

Instrument S/N can be read from the instrument, it is in the BootedPacketStruct and duplicated also in the ExtendedCalibrationCoefficients structure.

[shunt16]

Ah yeah makes sense you would have to get times from the instrument.

Perhaps @corizzi is already using this to populate the datetime and SN metadata anyway then? And latlon from GPS?

[jkuusk]

The spectral packets are dumped straight into .spe files together with packet headers. https://github.com/HYPERNETS/hypernets_processor/blob/master/hypernets_processor/data_io/spectrum.py

[clemgoyens]

@jkuusk just wondering, you are talking about the timestamp no? Cause I don't see the SN of the instrument within the packet headers of the .spe files