ets / tap-spreadsheets-anywhere

GNU Affero General Public License v3.0
31 stars 63 forks source link

Add JSONPath parser support #44

Open TyShkan opened 1 year ago

TyShkan commented 1 year ago

Closes https://github.com/ets/tap-spreadsheets-anywhere/issues/42

I struggled to run unittests due to dependencies conflicts, so I ran the test config with Meltano:

    config:
      tables:
        - "name": "json_sample_multiple_records"
          "path": "file://../../data/test/"
          "pattern": "^sample\\.json$"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "json"
        - "name": "json_sample_one_record"
          "path": "file://../../data/test/"
          "pattern": "^one-row-sample\\.json$"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "json"
        - "name": "jsonl_sample_multiple_records"
          "path": "file://../../data/test/"
          "pattern": "sample-jsonl\\.json"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "jsonl"
          "universal_newlines": True
        - "name": "jsonl_sample_one_record"
          "path": "file://../../data/test/"
          "pattern": "one-row-sample-jsonl\\.json"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "jsonl"
        - "name": "jsonl_sample_multiple_records_detect"
          "path": "file://../../data/test/"
          "pattern": "^sample\\.jsonl$"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "detect"
        - "name": "jsonl_sample_one_record_detect"
          "path": "file://../../data/test/"
          "pattern": "^one-row-sample\\.jsonl$"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "format": "detect"
        - "name": "json_sample_nested_path"
          "path": "file://../../data/test/"
          "pattern": "^sample_path\\.json$"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "json_path": "data"
          "format": "json"
        - "name": "json_sample_deep_nested_path"
          "path": "file://../../data/test/"
          "pattern": "sample_deep_path\\.json"
          "start_date": "2017-05-01T00:00:00Z"
          "key_properties": [ "id" ]
          "json_path": "data.response[*]"
          "format": "json"

with some testing files included: test.zip

jcbmllgn commented 1 year ago

Hello 👋 any update on when this might make it into main? This would be very useful for me!