search

NREL / h5plexos

Convert PLEXOS solutions to HDF5 and query results from Python
Other
5 stars 1 forks source link

RuntimeError: Unable to create link (name already exists) #7

Open MasonBowen opened 4 years ago

MasonBowen commented 4 years ago

Hi @GordStephen , I have been using the process_solution function from this package to convert PLEXOS solution files to an hdf5 format so that I can further process them for visualization efforts, etc.

For the first batch of files I had no difficulty using the function. I accessed the files on our shared drive (I can give you exact location over email). The first batch of files were for ST solutions, real time, for an India model. While long, the process worked well and all the files were successfully converted.

When I tried to run the same script for day ahead solutions (still ST, still India model) I got the following error:

Error

Traceback (most recent call last):

  File "<ipython-input-32-3199133c37b9>", line 7, in <module>
    process_solution(i, save_path)

  File "C:\Users\tbowen\Documents\Python\envs\h5plexos\lib\site-packages\h5plexos\process\buildh5.py", line 139, in process_solution
    int(length), int(offset))

  File "C:\Users\tbowen\Documents\Python\envs\h5plexos\lib\site-packages\h5plexos\process\metadata.py", line 30, in create_time_dset
    chunks=(length,), compression="gzip", compression_opts=1)

  File "C:\Users\tbowen\Documents\Python\envs\h5plexos\lib\site-packages\h5py\_hl\group.py", line 139, in create_dataset
    self[name] = dset

  File "C:\Users\tbowen\Documents\Python\envs\h5plexos\lib\site-packages\h5py\_hl\group.py", line 371, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)

  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper

  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper

  File "h5py\h5o.pyx", line 202, in h5py.h5o.link

RuntimeError: Unable to create link (name already exists)

My code is as follows:

    import os
    import pandas as pd
    import h5plexos as h5
    import re
    import numpy as np
    from h5plexos.process import process_solution

def extract(raw_string, start_marker, end_marker):
    start = raw_string.index(start_marker) + len(start_marker)
    end = raw_string.index(end_marker, start)
    return raw_string[start:end]

plexos_solutions_dir = r"I can share this folder with you"

def return_folder_locations(re_scen = "High Solar", rt_or_da = "RT", add_scen = "Reg Solution", solution_path = plexos_solutions_dir):

    pattern_match = f'^Model Regional {re_scen} [A-Z]{{2}} {rt_or_da} {add_scen}'

    solution_folders = os.listdir(plexos_solutions_dir)
    solution_folders_bool = [re.match(pattern_match,x) for x in solution_folders]

    solution_folders = [solution_folders[i] for i in np.where([bool(x) for x in solution_folders_bool])[0]]

    solution_order = [extract(x, f'Model Regional {re_scen} ', f' {rt_or_da} {add_scen}') for x in solution_folders]
    solution_order =  [solution_order.index(x) for x in ["JF", "MA", "MJ", "JA", "SO", "ND"]]

    solution_folders = [solution_folders[x] for x in solution_order]
    path_ret = [os.path.join(solution_path, x) for x in solution_folders]

    path_ret = [[os.path.join(y, x) for x in os.listdir(y) if x.endswith(".zip")][0] for y in path_ret]

    return(path_ret)

solution_paths = return_folder_locations(rt_or_da="DA", re_scen = "High Wind", add_scen = "St Solution")

save_dir = r"C:\Users\-----\Documents\USAID DPV\Value of Solar\PLEXOS Interaction"

for i in solution_paths:    

    save_name = os.path.basename(i).replace(" ", "_").replace(".zip", ".h5")
    save_path = os.path.join(save_dir, save_name)
    print(save_name)
    process_solution(i, save_path)

Here is the list of packages in my environment from which I'm running the script:

Package list

# Name                    Version                   Build  Channel
alabaster                 0.7.12                   py37_0
argh                      0.26.2                   py37_0
asn1crypto                1.2.0                    py37_0
astroid                   2.3.3                    py37_0
atomicwrites              1.3.0                    py37_1
attrs                     19.3.0                     py_0
autopep8                  1.4.4                      py_0
babel                     2.7.0                      py_0
backcall                  0.1.0                    py37_0
bcrypt                    3.1.7            py37he774522_0
blas                      1.0                         mkl
bleach                    3.1.0                    py37_0
ca-certificates           2019.11.27                    0
certifi                   2019.11.28               py37_0
cffi                      1.13.2           py37h7a1dbc1_0
chardet                   3.0.4                 py37_1003
cloudpickle               1.2.2                      py_0
colorama                  0.4.1                    py37_0
cryptography              2.8              py37h7a1dbc1_0
cython                    0.29.14          py37ha925a31_0
decorator                 4.4.1                      py_0
defusedxml                0.6.0                      py_0
diff-match-patch          20181111                   py_0
docutils                  0.15.2                   py37_0
entrypoints               0.3                      py37_0
flake8                    3.7.9                    py37_0
future                    0.18.2                   py37_0
h5plexos                  0.4.2                    pypi_0    pypi
h5py                      2.9.0            py37h5e291fa_0
hdf5                      1.10.4               h7ebc959_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.8                      py37_0
imagesize                 1.1.0                    py37_0
importlib_metadata        1.1.0                    py37_0
intel-openmp              2019.4                      245
intervaltree              3.0.2                      py_0
ipykernel                 5.1.3            py37h39e3cac_0
ipython                   7.10.1           py37h39e3cac_0
ipython_genutils          0.2.0                    py37_0
isort                     4.3.21                   py37_0
jedi                      0.14.1                   py37_0
jinja2                    2.10.3                     py_0
jpeg                      9b                   hb83a4c4_2
jsonschema                3.2.0                    py37_0
jupyter_client            5.3.4                    py37_0
jupyter_core              4.6.1                    py37_0
keyring                   19.2.0                   py37_0
lazy-object-proxy         1.4.3            py37he774522_0
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.16               h9d3ae62_0
libspatialindex           1.9.3                h33f27b4_0
markupsafe                1.1.1            py37he774522_0
mccabe                    0.6.1                    py37_1
mistune                   0.8.4            py37he774522_0
mkl                       2019.4                      245
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.15           py37h14836fe_0
mkl_random                1.1.0            py37h675688f_0
more-itertools            7.2.0                    py37_0
nbconvert                 5.6.1                    py37_0
nbformat                  4.4.0                    py37_0
numpy                     1.17.4           py37h4320e6b_0
numpy-base                1.17.4           py37hc3f5095_0
numpydoc                  0.9.1                      py_0
openssl                   1.1.1d               he774522_3
packaging                 19.2                       py_0
pandas                    0.25.3           py37ha925a31_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py37_1
paramiko                  2.6.0                    py37_0
parso                     0.5.1                      py_0
pathtools                 0.1.2                      py_1
pexpect                   4.7.0                    py37_0
pickleshare               0.7.5                    py37_0
pip                       19.3.1                   py37_0
pluggy                    0.13.1                   py37_0
prompt_toolkit            3.0.2                      py_0
psutil                    5.6.7            py37he774522_0
pycodestyle               2.5.0                    py37_0
pycparser                 2.19                     py37_0
pydocstyle                4.0.1                      py_0
pyflakes                  2.1.1                    py37_0
pygments                  2.5.2                      py_0
pylint                    2.4.4                    py37_0
pympler                   0.7                        py_0
pynacl                    1.3.0            py37h62dcd97_0
pyopenssl                 19.1.0                   py37_0
pyparsing                 2.4.5                      py_0
pyqt                      5.9.2            py37h6538335_2
pyreadline                2.1                      py37_1
pyrsistent                0.15.6           py37he774522_0
pysocks                   1.7.1                    py37_0
python                    3.7.5                h8c8aaf0_0
python-dateutil           2.8.1                      py_0
python-jsonrpc-server     0.3.2                      py_0
python-language-server    0.31.2                   py37_0
pytz                      2019.3                     py_0
pywin32                   223              py37hfa6e2cd_1
pywin32-ctypes            0.2.0                    py37_0
pyyaml                    5.1.2            py37he774522_0
pyzmq                     18.1.0           py37ha925a31_0
qdarkstyle                2.7                        py_0
qt                        5.9.7            vc14h73c81de_0
qtawesome                 0.6.0                      py_0
qtconsole                 4.6.0                      py_0
qtpy                      1.9.0                      py_0
requests                  2.22.0                   py37_1
rope                      0.14.0                     py_0
rtree                     0.8.3                    py37_0
setuptools                42.0.2                   py37_0
sip                       4.19.8           py37h6538335_0
six                       1.13.0                   py37_0
snowballstemmer           2.0.0                      py_0
sortedcontainers          2.1.0                    py37_0
sphinx                    2.2.2                      py_0
sphinxcontrib-applehelp   1.0.1                      py_0
sphinxcontrib-devhelp     1.0.1                      py_0
sphinxcontrib-htmlhelp    1.0.2                      py_0
sphinxcontrib-jsmath      1.0.1                      py_0
sphinxcontrib-qthelp      1.0.2                      py_0
sphinxcontrib-serializinghtml 1.1.3                      py_0
spyder                    4.0.0                    py37_0
spyder-kernels            1.8.1                    py37_0
sqlite                    3.30.1               he774522_0
testpath                  0.4.4                      py_0
tornado                   6.0.3            py37he774522_0
traitlets                 4.3.3                    py37_0
ujson                     1.35             py37hfa6e2cd_0
urllib3                   1.25.7                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_0
watchdog                  0.9.0                    py37_1
wcwidth                   0.1.7                    py37_0
webencodings              0.5.1                    py37_1
wheel                     0.33.6                   py37_0
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
wrapt                     1.11.2           py37he774522_0
yaml                      0.1.7                hc54c509_2
yapf                      0.28.0                     py_0
zeromq                    4.3.1                h33f27b4_3
zipp                      0.6.0                      py_0
zlib                      1.2.11               h62dcd97_3

Any ideas as to what could be wrong? Please let me know what other information I should share with you. Thanks!

GordStephen commented 4 years ago

My guess would be that the DA results include MT outputs (see #4 and #6) - @mattirish is going to take a shot at fixing this but for now the immediate workaround is to not report out MT data.

MasonBowen commented 4 years ago

Hi Gord, Thanks for the insight!,For solutions that already exist is there any way around this?

Thanks!

GordStephen commented 4 years ago

In a pinch, it may be possible to do some surgery on the XML file contents inside the zipfile produced by PLEXOS, eliminating the references to the MT results, but that's unexplored territory - if at all possible I would just re-run with a modified report object.