search

khaeru / sdmx

SDMX information model and client in Python
https://sdmx1.readthedocs.io
Apache License 2.0
24 stars 18 forks source link

OECD dataflow RuntimeError #155

Closed leostimpfle closed 8 months ago

leostimpfle commented 8 months ago

Using the OECD client, I receive a RuntimeError when using the dataflow method. The data method seems to work fine (unless the key kwarg is specified). I'm using sdmx version '2.12.1' on Python '3.12.1'.

Code

import sdmx

source = 'OECD'
flow = 'DSD_CAPMF@DF_CAPMF'
client = sdmx.Client(source)

response = client.data(flow)  # this works
dsd = client.dataflow(flow)  # this throws RuntimeError
response = client.data(flow, key=dict(REF_AREA=['DEU'])  # this throws RuntimeError

Output

--- SS without DSD ---
1 (4359034456) False

--- <class 'sdmx.message.StructureMessage'> ---
2 (12870790864) <sdmx.StructureMessage>
  <Header>
    id: 'IDREF1408'
    prepared: '2024-01-12T11:12:17.275721+01:00'
    receiver: <Agency Unknown>
    sender: <Agency Unknown>
    source: 
    test: False

--- <class 'sdmx.model.v21.DataStructureDefinition'> ---
14136 (12826220048) DSD_CAPMF
14137 (12826220048) DSD_CAPMF

--- <class 'sdmx.model.common.Agency'> ---
OECD.ENV.EPI (12826219616) OECD.ENV.EPI
OECD (12826013536) OECD
SDMX (12882838688) SDMX

--- <class 'sdmx.model.v21.DataflowDefinition'> ---
DSD_CAPMF@DF_CAPMF (12826220432) DSD_CAPMF@DF_CAPMF

--- <class 'sdmx.model.common.CategoryScheme'> ---
914 (12826013680) OECDCS1
915 (12826014256) OECDCS1

--- <class 'sdmx.model.common.Categorisation'> ---
CAT_OECD_ENV_EPI__DSD_CAPMF@DF_CAPMF__1_0@OECD__OECDCS1__2_0__ENV_ENV_POL (12826015072) CAT_OECD_ENV_EPI__DSD_CAPMF@DF_CAPMF__1_0@OECD__OECDCS1__2_0__ENV_ENV_POL

--- <class 'sdmx.model.common.Codelist'> ---
CL_AREA (12896637328) CL_AREA
CL_UNIT_MEASURE (12884887840) CL_UNIT_MEASURE
CL_ACT_POL_CAPMF (12883046816) CL_ACT_POL_CAPMF
CL_MEASURE_CAPMF (12883047632) CL_MEASURE_CAPMF
CL_DECIMALS (12882839024) CL_DECIMALS
CL_FREQ (12882850832) CL_FREQ
CL_OBS_STATUS (12882692192) CL_OBS_STATUS
CL_UNIT_MULT (12882701072) CL_UNIT_MULT

--- <class 'sdmx.model.common.ConceptScheme'> ---
14148 (12882704960) CS_CAPMF
14149 (12882705776) CS_CAPMF
CS_CAPMF (12882592064) CS_CAPMF

--- <class 'sdmx.model.common.Annotation'> ---
14205 (12882705200) Annotation(id=None, title='A', type='ReleaseVersion', url=None, text=)
obs_count (12826220240) Annotation(id='obs_count', title='380952', type='sdmx_metrics', url=None, text=)

--- Name ---
14209 (12847676096) ('en', 'Availability (A) for DSD_CAPMF@DF_CAPMF')

--- <class 'sdmx.reader.xml.v21.Reference'> ---
DSD_CAPMF@DF_CAPMF (12882596528) DataflowDefinition=OECD.ENV.EPI:DSD_CAPMF@DF_CAPMF(1.0) → DataflowDefinition=DSD_CAPMF@DF_CAPMF

--- <class 'sdmx.model.v21.MemberSelection'> ---
14263 (12882593312) <MemberSelection REF_AREA in {'ARG', 'AUS', 'AUT', 'BEL', 'BGR', 'BRA', 'CAN', 'CHE', 'CHL', 'CHN', 'COL', 'CRI', 'CZE', 'DEU', 'DNK', 'ESP', 'EST', 'EU', 'FIN', 'FRA', 'GBR', 'GRC', 'HRV', 'HUN', 'IDN', 'IND', 'IRL', 'ISL', 'ISR', 'ITA', 'JPN', 'KOR', 'LTU', 'LUX', 'LVA', 'MEX', 'MLT', 'NLD', 'NOR', 'NZL', 'PER', 'POL', 'PRT', 'ROU', 'RUS', 'SAU', 'SVK', 'SVN', 'SWE', 'TUR', 'USA', 'ZAF'}>
14265 (12882600944) <MemberSelection FREQ in {'A'}>
14268 (12882593888) <MemberSelection MEASURE in {'POL_COUNT', 'POL_STRINGENCY'}>
14473 (12882601232) <MemberSelection CLIM_ACT_POL in {'LEV1_CROSS_SEC', 'LEV1_INT', 'LEV1_SEC', 'LEV2_CROSS_SEC_CG', 'LEV2_CROSS_SEC_FFPP', 'LEV2_CROSS_SEC_GHGTAR', 'LEV2_CROSS_SEC_RDD', 'LEV2_INT_C_COORD', 'LEV2_INT_C_FIN', 'LEV2_INT_GHGREP', 'LEV2_SEC_B_MBI', 'LEV2_SEC_B_NMBI', 'LEV2_SEC_E_MBI', 'LEV2_SEC_E_NMBI', 'LEV2_SEC_I_MBI', 'LEV2_SEC_I_NMBI', 'LEV2_SEC_T_MBI', 'LEV2_SEC_T_NMBI', 'LEV3_AB', 'LEV3_AUCTION', 'LEV3_BAN_CREDIT', 'LEV3_BAN_FF_ABROAD', 'LEV3_BAN_PHOUT_COAL', 'LEV3_BAN_PHOUT_EXTRAC', 'LEV3_BAN_PHOUT_HEAT', 'LEV3_BAN_PHOUT_ICE', 'LEV3_BC', 'LEV3_CARBONTAX_B', 'LEV3_CARBONTAX_E', 'LEV3_CARBONTAX_I', 'LEV3_CARBONTAX_T', 'LEV3_CONG_CHARG', 'LEV3_EE_MANDATE', 'LEV3_EMIS_STD', 'LEV3_ETS_B', 'LEV3_ETS_E', 'LEV3_ETS_I', 'LEV3_ETS_T', 'LEV3_EVAL_BR', 'LEV3_EXCISETAX_B', 'LEV3_EXCISETAX_E', 'LEV3_EXCISETAX_I', 'LEV3_EXCISETAX_T', 'LEV3_EXP_RAIL', 'LEV3_FFS_B', 'LEV3_FFS_E', 'LEV3_FFS_I', 'LEV3_FFS_PRODUCER', 'LEV3_FFS_T', 'LEV3_FIN_MECH_B', 'LEV3_FIN_MECH_I', 'LEV3_FIT', 'LEV3_GHG_ACC', 'LEV3_INT_INIT', 'LEV3_LABEL_APPL', 'LEV3_LABEL_CAR', 'LEV3_MEPS_APPL', 'LEV3_MEPS_MOTOR', 'LEV3_MEPS_T', 'LEV3_METHAN', 'LEV3_NDC', 'LEV3_NZ', 'LEV3_PR_AV_MAR', 'LEV3_RDD_CCS', 'LEV3_RDD_EE', 'LEV3_RDD_HYDR', 'LEV3_RDD_NUC', 'LEV3_RDD_OTHER', 'LEV3_RDD_RES', 'LEV3_RECS', 'LEV3_RENEWABLE_EXP', 'LEV3_SPEED', 'LEV3_TREATY', 'LEV3_UNFCCC', 'LEV4_AB_BUDGET', 'LEV4_AB_COUNC_MEM', 'LEV4_AB_ESTABL', 'LEV4_AB_ESTABL_LAW', 'LEV4_AB_SECR_MEM', 'LEV4_AUCTION_SOL_DUR', 'LEV4_AUCTION_SOL_PR', 'LEV4_AUCTION_WIND_DUR', 'LEV4_AUCTION_WIND_PR', 'LEV4_BAN_COAL_DATE', 'LEV4_BAN_COAL_STAT', 'LEV4_BAN_CREDIT_DATE', 'LEV4_BAN_CREDIT_STAT', 'LEV4_BAN_EXTRAC_DATE', 'LEV4_BAN_EXTRAC_STAT', 'LEV4_BAN_FF_ABROAD_DATE', 'LEV4_BAN_FF_ABROAD_STAT', 'LEV4_BAN_GAS_DATE', 'LEV4_BAN_GAS_STAT', 'LEV4_BAN_ICE_DATE', 'LEV4_BAN_ICE_STAT', 'LEV4_BAN_OIL_DATE', 'LEV4_BAN_OIL_STAT', 'LEV4_BC_NON_RES', 'LEV4_BC_RES', 'LEV4_CARBONTAX_B', 'LEV4_CARBONTAX_E_PR', 'LEV4_CARBONTAX_I', 'LEV4_CARBONTAX_T', 'LEV4_CONG_CHARG', 'LEV4_EE_MANDATE', 'LEV4_EMIS_STD_NOX', 'LEV4_EMIS_STD_PM', 'LEV4_EMIS_STD_SOX', 'LEV4_ETS_B_GHG', 'LEV4_ETS_B_PR', 'LEV4_ETS_E_GHG', 'LEV4_ETS_E_PR', 'LEV4_ETS_I_GHG', 'LEV4_ETS_I_PR', 'LEV4_ETS_T_GHG', 'LEV4_ETS_T_PR', 'LEV4_EVAL_BR', 'LEV4_EXCISETAX_B_C_COAL', 'LEV4_EXCISETAX_B_C_DIESEL', 'LEV4_EXCISETAX_B_C_LPG', 'LEV4_EXCISETAX_B_C_NATGAS', 'LEV4_EXCISETAX_B_R_COAL', 'LEV4_EXCISETAX_B_R_DIESEL', 'LEV4_EXCISETAX_B_R_LPG', 'LEV4_EXCISETAX_B_R_NATGAS', 'LEV4_EXCISETAX_E_COAL', 'LEV4_EXCISETAX_E_NATGAS', 'LEV4_EXCISETAX_I_COAL', 'LEV4_EXCISETAX_I_DIESEL', 'LEV4_EXCISETAX_I_NATGAS', 'LEV4_EXCISETAX_T_DIESEL', 'LEV4_EXCISETAX_T_GASO', 'LEV4_EXP_RAIL', 'LEV4_FFS_B', 'LEV4_FFS_E', 'LEV4_FFS_I', 'LEV4_FFS_PRODUCER', 'LEV4_FFS_T', 'LEV4_FIN_MECH_B_C', 'LEV4_FIN_MECH_B_R', 'LEV4_FIN_MECH_I', 'LEV4_FIT_SOL_DUR', 'LEV4_FIT_SOL_PR', 'LEV4_FIT_WIND_DUR', 'LEV4_FIT_WIND_PR', 'LEV4_GHG_ACC_ANNEX', 'LEV4_GHG_ACC_SEEA', 'LEV4_INT_INIT', 'LEV4_LABEL_AC', 'LEV4_LABEL_CAR', 'LEV4_LABEL_FRE', 'LEV4_LABEL_LIGHT', 'LEV4_LABEL_REF', 'LEV4_MEPS_AC', 'LEV4_MEPS_FRE', 'LEV4_MEPS_HDV', 'LEV4_MEPS_LDV', 'LEV4_MEPS_LIGHT', 'LEV4_MEPS_MOTOR', 'LEV4_MEPS_REF', 'LEV4_METHAN', 'LEV4_NDC_DET', 'LEV4_NDC_LULUCF', 'LEV4_NDC_SCOPE_EMIS', 'LEV4_NDC_SCOPE_SEC', 'LEV4_NDC_SING_MULT', 'LEV4_NDC_TARG_TYPE', 'LEV4_NZ_DATE', 'LEV4_NZ_GHG_SCOPE', 'LEV4_NZ_SEC_SCOPE', 'LEV4_NZ_STAT', 'LEV4_PHOUT_COAL_DATE', 'LEV4_PHOUT_COAL_STAT', 'LEV4_PHOUT_EXTRAC_DATE', 'LEV4_PHOUT_EXTRAC_STAT', 'LEV4_PHOUT_GAS_DATE', 'LEV4_PHOUT_GAS_STAT', 'LEV4_PHOUT_ICE_DATE', 'LEV4_PHOUT_ICE_STAT', 'LEV4_PHOUT_OIL_DATE', 'LEV4_PHOUT_OIL_STAT', 'LEV4_PR_AV', 'LEV4_PR_CORSIA', 'LEV4_PR_MAR', 'LEV4_RDD_CCS', 'LEV4_RDD_EE', 'LEV4_RDD_HYDR', 'LEV4_RDD_NUC', 'LEV4_RDD_OTHER', 'LEV4_RDD_RES', 'LEV4_RECS', 'LEV4_RENEWABLE_EXP', 'LEV4_SPEED', 'LEV4_TREATY_KIGALI', 'LEV4_TREATY_KYOTO', 'LEV4_TREATY_MONTREALAMEND', 'LEV4_TREATY_MONTREALPROT', 'LEV4_TREATY_PARIS', 'LEV4_TREATY_UNFCCC', 'LEV4_UNFCCC_BR', 'LEV4_UNFCCC_GHGINV', 'LEV4_UNFCCC_LTLEDS', 'LEV4_UNFCCC_NC', 'LEV4_UNFCCC_NDC'}>
14476 (12883266448) <MemberSelection UNIT_MEASURE in {'0_TO_10', 'PL'}>

--- <class 'sdmx.model.v21.RangePeriod'> ---
14479 (12883266880) RangePeriod(start=Period(is_inclusive='true', period=datetime.datetime(1990, 1, 1, 0, 0)), end=Period(is_inclusive='true', period=datetime.datetime(2022, 12, 31, 0, 0)))

Ignore:
 {4358957264, 12826013536, 12826219616, 12882838688}
<common:KeyValue xmlns:common="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common" xmlns:message="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:structure="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/structure" id="TIME_PERIOD">
            <common:TimeRange/></common:KeyValue>

Traceback (most recent call last):

  File ~/Documents/code/venvs/competition-venv/lib/python3.12/site-packages/sdmx/reader/xml/v21.py:274 in read_message
    result = func(self, element)

  File ~/Documents/code/venvs/competition-venv/lib/python3.12/site-packages/sdmx/reader/xml/v21.py:1368 in _ms
    raise RuntimeError

RuntimeError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  Cell In[26], line 5
    dsd = client.dataflow(flow)

  File ~/Documents/code/venvs/competition-venv/lib/python3.12/site-packages/sdmx/client.py:486 in get
    msg = reader.read_message(response_content, dsd=kwargs.get("dsd", None))

  File ~/Documents/code/venvs/competition-venv/lib/python3.12/site-packages/sdmx/reader/xml/__init__.py:45 in read_message
    import_module(f"sdmx.reader.xml.{version}")

  File ~/Documents/code/venvs/competition-venv/lib/python3.12/site-packages/sdmx/reader/xml/v21.py:291 in read_message
    raise XMLParseError from exc

XMLParseError: RuntimeError

Versions

import sys
sys.version
3.12.1 (main, Dec  7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.0.40.1)
sdmx.__version__
'2.12.1'
khaeru commented 8 months ago

Thanks for the report!

I suspect this may have been fixed here by a commit from #152, which is not yet released.

Are you able to try installing from the main branch here, to confirm whether the bug still occurs with the most recent code? For instance, you could do:

pip install git+https://github.com/khaeru/sdmx.git@main

(Documentation at https://pip.pypa.io/en/stable/topics/vcs-support/)

If not, please just say so and I will try myself to confirm.

leostimpfle commented 8 months ago

Thank you for the quick response! Everything is working using the main branch.

khaeru commented 8 months ago

Great, thanks for confirming! After #154 is complete I will release v2.13, and then the fix should be available in the version available from PyPI.