search

cedadev / item-generator

0 stars 2 forks source link

asset-scanner tries to run on everything in the directory so it fails #21

Open ellesmith88 opened 3 years ago

ellesmith88 commented 3 years ago

I tried to run asset_scanner on a description for file for /neodc/sentinel5p/data/L1_RA/v1.0/2020/01/01/ but it tried to run on .ftpaccess in the same directory as the netCDF files.

2021-07-02 11:54:54,738 item_generator.core.facet_extractor INFO Processing: /neodc/sentinel5p/data/L1_RA/v1.0/2020/01/01/.ftpaccess
Traceback (most recent call last):
  File "/home/users/esmith88/search/venv/bin/asset_scanner", line 11, in <module>
    load_entry_point('asset-scanner', 'console_scripts', 'asset_scanner')()
  File "/home/users/esmith88/search/asset-scanner/asset_scanner/scripts/asset_scanner.py", line 90, in main
    input.run(extractor)
  File "/home/users/esmith88/search/asset-scanner/asset_scanner/plugins/input_plugins/file_system_input.py", line 54, in run
    extractor.process_file(filename, 'posix')
  File "/home/users/esmith88/search/item-generator/item_generator/core/facet_extractor.py", line 146, in process_file
    tags = self.get_facets(filepath, description, source_media)
  File "/home/users/esmith88/search/item-generator/item_generator/core/facet_extractor.py", line 114, in get_facets
    metadata = p.run(filepath, source_media=source_media, post_processors=post_processors, pre_processors=pre_processors)
  File "/home/users/esmith88/search/item-generator/item_generator/core/decorators.py", line 66, in wrapper
    response = func(*args, **kwargs)
TypeError: run() missing 1 required positional argument: 'attributes'
agstephens commented 3 years ago

A good solution would be to define a set of file/dir/symlink name patterns that should be ignored by the extractors.