ets / tap-spreadsheets-anywhere

GNU Affero General Public License v3.0
31 stars 63 forks source link

Walking a non existant local file directory doesn't fail #47

Closed visch closed 10 months ago

visch commented 1 year ago
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/spreadsheets$ dir
README.md  analyze  config.json  extract  load  meltano.yml  notebook  orchestrate  output  plugins  requirements.txt  transform
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/spreadsheets$ cat meltano.yml
version: 1
default_environment: dev
project_id: 676da67f-9ef4-44b9-8993-fcf9513de2af
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
      - path: file://extractttt/
        name: d11_test
        pattern: "/*.csv"
        start_date: '2023-01-01T00:00:00Z'
        key_properties: [Unique Record ID]
        format: csv
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/spreadsheets$ meltano inv^C
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/spreadsheets$ meltano invoke tap-spreadsheets-anywhere
2023-03-29T19:19:47.436312Z [info     ] Environment 'dev' is active
INFO Using supplied catalog /home/visch/git/meltano-projects/spreadsheets/.meltano/run/tap-spreadsheets-anywhere/tap.properties.json.
INFO Processing 1 selected streams from Catalog
INFO Syncing stream:d11_test
{"type": "SCHEMA", "stream": "d11_test", "schema": {"properties": {"_smart_source_bucket": {"type": "string"}, "_smart_source_file": {"type": "string"}, "_smart_source_lineno": {"type": "integer"}}, "selected": true, "type": "object"}, "key_properties": ["Unique Record ID"]}
INFO Walking extractttt/.
INFO Found 0 files.
INFO Checking 0 resolved objects for any that match regular expression "/*.csv" and were modified since 2023-01-01 00:00:00+00:00
INFO Processing 0 resolved objects that met our criteria. Enable debug verbosity logging for more details.
INFO Wrote 0 records for stream "d11_test".
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/spreadsheets$

This should fail in my opinion instead of running and not telling you something is wrong. I'm trying to pull data from a drive on windows and it's really hard to tell why I can't see the files in the directory.