search

NREL / h5plexos

Convert PLEXOS solutions to HDF5 and query results from Python
Other
5 stars 1 forks source link

Fix error in process_solution coming from h5py's create_dataset #6

Open mattirish opened 5 years ago

mattirish commented 5 years ago

I'm just getting this error when trying to process a PLEXOS solution from RPM-LA. I've successfully used h5plexos to process the test solution and our NAERM RTS-GMLC network solutions, so there aren't any issues with my installation. I get this same error for any year and scenario of our RPM-LA runs, and with runs from a few months back, too:

>>> process_solution("Model base_2020_h1 Solution.zip","2020_h1.h5")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mirish/anaconda3/lib/python3.6/site-packages/h5plexos-0.4.2-py3.6.egg/h5plexos/process/buildh5.py", line 139, in process_solution
  File "/Users/mirish/anaconda3/lib/python3.6/site-packages/h5plexos-0.4.2-py3.6.egg/h5plexos/process/metadata.py", line 30, in create_time_dset
  File "/Users/mirish/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 109, in create_dataset
    self[name] = dset
  File "/Users/mirish/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 277, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

An hdf5 database is created with correct metadata but no data.

From a quick googling, it looks like h5py throws this error in create_dataset when we're trying to create an already existing dataset.

It's hard to know right off that bat what in our PLEXOS representation of RPM is causing this. Will try to figure it out!

@GordStephen, the solution being processed above is in //nrelqnap02/RPM/Projects/LA100/rpm_la/20190523_la_sb100/islanded, if ya want to check it out. The xml is at 20190523_la_sb100/inout/plexos_output/final_20190523_la_sb100_is.xml.

GordStephen commented 5 years ago

Pretty sure this is #4 - can you try turning off MT reporting and see if that fixes it?

mattirish commented 5 years ago

Ah, sorry! I glanced at the existing issues but it didn't sound related from the title. I'll set off a run without MT reporting tonight and let ya know. Thanks.

GordStephen commented 5 years ago

@mattirish, did disabling MT reporting fix this for you?

GordStephen commented 4 years ago

@mattirish, following up on our conversation today, if you're interested in taking a stab at this, I think this is the offending code: https://github.com/NREL/h5plexos/blob/33dc07a6b1ffe47574ad5cf73c5243584508a15d/h5plexos/process/buildh5.py#L140-L142

Basically, the MT timestamp data in the PLEXOS file must have a period_num that conflicts with the key defined here: https://github.com/NREL/h5plexos/blob/33dc07a6b1ffe47574ad5cf73c5243584508a15d/h5plexos/process/buildh5.py#L22-L28

It may be that different phases have overlapping period_nums and that some other column in the database is needed to disambiguate (e.g. there might be an ST 0 and an MT 0) - h5plexos ends up trying to create a 0="interval" dataset twice which leads to the RuntimeError: Unable to create link (name already exists)