search

OasisLMF / ODS_Tools

Python package supporting ODS OpenDataStandards (OED / ORD)
3 stars 1 forks source link

Package does not include the validation files #151

Open carlfischerjba opened 1 month ago

carlfischerjba commented 1 month ago

Issue Description

Running ods_tools transform displays warnings about missing validation files.

Steps to Reproduce (Bugs only)

Command:

ods_tools transform --config-file /workspace/LM3932_cede_conversion/CF_config.yaml --nocheck
transformations:
  loc: # Transformation name
    input_format:
      name: Cede_Location
      version: 10.0.0
    output_format:
      name: OED_Location
      version: 3.0.2
    runner:
      batch_size: 750000 # Number of rows to process in a single batch
    extractor:
      type: mssql # Path to the DB connection file. Other options are 'postgres' and 'sqlite'. Defaults to 'csv', when not present.
      options:
        host: myserver,1450
        port:
        database: MY_DB
        user: foo
        password: bar
        sql_statement: ./cede_location_1000.sql # SQL statement to extract data
    loader:
      options:
        path: ./CF_oed_location.csv # Path to the output file
        quoting: minimal

Error:

2024-10-28 16:57:47,684 - WARNING - Could not find validator config for Cede_Location. Tried paths /workspace/LM3932_cede_conversion/validation_Cede_Location_v10.0.0_loc.yaml, /home/carl/venvs/myvenv/lib/python3.10/site-packages/ods_tools/odtf/data/validators/validation_Cede_Location_v10.0.0_loc.yaml, /workspace/LM3932_cede_conversion/validation_Cede_Location_loc.yaml, /home/carl/venvs/myvenv/lib/python3.10/site-packages/ods_tools/odtf/data/validators/validation_Cede_Location_loc.yaml
2024-10-28 16:57:48,079 - WARNING - Could not find validator config for OED_Location. Tried paths /workspace/LM3932_cede_conversion/validation_OED_Location_v3.0.2_loc.yaml, /home/carl/venvs/myvenv/lib/python3.10/site-packages/ods_tools/odtf/data/validators/validation_OED_Location_v3.0.2_loc.yaml, /workspace/LM3932_cede_conversion/validation_OED_Location_loc.yaml, /home/carl/venvs/myvenv/lib/python3.10/site-packages/ods_tools/odtf/data/validators/validation_OED_Location_loc.yaml

I think the following error may also be caused by this:

2024-10-28 16:51:47,903 - ERROR - VisitError('Error trying to process rule "lookup":\n\nToken(\'IDENT\', \'True\')'), line 160 in /home/carl/venvs/myvenv/lib/python3.10/site-packages/ods_tools/odtf/controller.py

Version / Environment information

Example data / logs

The issue disappears if we clone the source code from this repo and run pip install --editable ., suggesting that the package description might need adjusting.

https://github.com/OasisLMF/ODS_Tools/blob/02c9e22172852c387f98e58a0d990b258cd0b96e/setup.py#L105

Might need to become "ods_tools": ["data/*", "odtf/data/mappings/*", "odtf/data/validators/*"].