S3 Delta adapter is failing with 'S3DeltaPlugin' object has no attribute 'dataset_config'

[dfustes]

After running the latest version of the odd-collector-aws locally, I've created this configuration:

plugins:
  - type: s3_delta
    name: s3_adapter
    aws_region: eu-west-1
    delta_tables:
      - bucket: <bucket>
        prefix: <prefix>

, but when I start the collector it's throwing this exception:

docker-odd-collector-aws-1  | 2024-06-19 14:01:59.993 | INFO     | odd_collector_sdk.domain.collector_config:load_config:37 - Start reading config
docker-odd-collector-aws-1  | Traceback (most recent call last):
docker-odd-collector-aws-1  |   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
docker-odd-collector-aws-1  |     return _run_code(code, main_globals, None,
docker-odd-collector-aws-1  |   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
docker-odd-collector-aws-1  |     exec(code, run_globals)
docker-odd-collector-aws-1  |   File "/app/odd_collector_aws/__main__.py", line 16, in <module>
docker-odd-collector-aws-1  |     collector = Collector(
docker-odd-collector-aws-1  |   File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/collector.py", line 65, in __init__
docker-odd-collector-aws-1  |     self._adapters = load_adapters(
docker-odd-collector-aws-1  |   File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/load_adapter.py", line 54, in load_adapters
docker-odd-collector-aws-1  |     adapter = package.adapter.Adapter(plugin)
docker-odd-collector-aws-1  |   File "/app/odd_collector_aws/adapters/s3_delta/adapter.py", line 23, in __init__
docker-odd-collector-aws-1  |     super().__init__(config)
docker-odd-collector-aws-1  |   File "/usr/local/lib/python3.9/site-packages/odd_collector_sdk/domain/adapter.py", line 26, in __init__
docker-odd-collector-aws-1  |     self.generator = self.create_generator()
docker-odd-collector-aws-1  |   File "/app/odd_collector_aws/adapters/s3_delta/adapter.py", line 27, in create_generator
docker-odd-collector-aws-1  |     return create_generator(S3Generator, self.config)
docker-odd-collector-aws-1  |   File "/app/odd_collector_aws/utils/create_generator.py", line 20, in create_generator
docker-odd-collector-aws-1  |     bucket = config.dataset_config.bucket
docker-odd-collector-aws-1  | AttributeError: 'S3DeltaPlugin' object has no attribute 'dataset_config'

, it seems to be the typical missing object reference error in Python. Could you take a look?

Thanks!

[ValeriyWorld]

We are investigating this problem.

[ValeriyWorld]

Fixed with #88