search

MeltanoLabs / tap-postgres

Singer Tap for PostgreSQL
https://hub.meltano.com/extractors/tap-postgres--meltanolabs/
Other
19 stars 23 forks source link

Faker not available in stream api #493

Closed maxmarcon closed 2 weeks ago

maxmarcon commented 2 weeks ago

I would like to add a stream_maps -based transformation and use faker in it. The transformation fails because "'fake' is not defined" . Here's my stream_maps config:

 config:
    stream_maps:
       data_platform_raw-tbii_daily_gmv_nmv:
           at_channel_id: fake.pyint()
       faker_config:
           seed: 0

And here's the error's stacktrace:

2024-09-02 15:20:38,215 | ERROR    | tap-postgres.data_platform_raw-tbii_daily_gmv_nmv | An unhandled error occurred while syncing 'data_platform_raw-tbii_daily_gmv_nmv'
Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 554, in _eval_name
    return self.names[node.id]
KeyError: 'fake'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 350, in _eval
    result: str | int | float = self.expr_evaluator.eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 664, in eval
    return super(EvalWithCompoundTypes, self).eval(expr, previously_parsed)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 428, in eval
    return self._eval(previously_parsed or self.parse(expr))
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 443, in _eval_expr
    return self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 527, in _eval_call
    func = self._eval(node.func)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 589, in _eval_attribute
    node_evaluated = self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 567, in _eval_name
    raise NameNotDefined(node.id, self.expr)
simpleeval.NameNotDefined: 'fake' is not defined for expression 'fake.pyint()'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1214, in sync
    for _ in self._sync_records(context=context):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1139, in _sync_records
    self._write_record_message(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 889, in _write_record_message
    for record_message in self._generate_record_messages(record):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 873, in _generate_record_messages
    mapped_record = stream_map.transform(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 285, in transform
    transformed_record = self._transform_fn(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 596, in transform_fn
    result[prop_key] = self._eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 356, in _eval
    raise MapExpressionError(msg) from ex
singer_sdk.exceptions.MapExpressionError: Failed to evaluate simpleeval expressions fake.pyint().
Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 554, in _eval_name
    return self.names[node.id]
KeyError: 'fake'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 350, in _eval
    result: str | int | float = self.expr_evaluator.eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 664, in eval
    return super(EvalWithCompoundTypes, self).eval(expr, previously_parsed)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 428, in eval
    return self._eval(previously_parsed or self.parse(expr))
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 443, in _eval_expr
    return self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 527, in _eval_call
    func = self._eval(node.func)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 589, in _eval_attribute
    node_evaluated = self._eval(node.value)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 440, in _eval
    return handler(node)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/simpleeval.py", line 567, in _eval_name
    raise NameNotDefined(node.id, self.expr)
simpleeval.NameNotDefined: 'fake' is not defined for expression 'fake.pyint()'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/plugin_base.py", line 82, in invoke
    return super().invoke(ctx)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 507, in invoke
    tap.sync_all()
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 466, in sync_all
    stream.sync()
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1214, in sync
    for _ in self._sync_records(context=context):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 1139, in _sync_records
    self._write_record_message(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 889, in _write_record_message
    for record_message in self._generate_record_messages(record):
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/streams/core.py", line 873, in _generate_record_messages
    mapped_record = stream_map.transform(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 285, in transform
    transformed_record = self._transform_fn(record)
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 596, in transform_fn
    result[prop_key] = self._eval(
  File "/Users/mmarcon/tb-ii/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/mapper.py", line 356, in _eval
    raise MapExpressionError(msg) from ex
singer_sdk.exceptions.MapExpressionError: Failed to evaluate simpleeval expressions fake.pyint().

The docs say:

Config for the Faker instance variable fake used within map expressions. Only applicable if the plugin specifies faker as an addtional dependency (through the singer-sdk faker extra or directly).

I tried installing the faker extra dep:

- name: tap-postgres
    variant: meltanolabs
    pip_url: meltanolabs-tap-postgres[faker] git+https://github.com/MeltanoLabs/tap-postgres.git@v0.0.11

But I get the same error.

How can I access faker for the stream_maps transformation with tap-postgres?

Meltano: 3.4.1 Python: 3.9.18

edgarrmondragon commented 2 weeks ago

v0.0.12 was published

edgarrmondragon commented 2 weeks ago

Related:

maxmarcon commented 2 weeks ago

Thanks for publishing the new version. I tried 0.0.12 but I'm unfortunately getting the same error, nothing changed 😞

Not a meltano or poetry expert by any means, but shouldn't the faker extra dep be listed here perhaps?

edgarrmondragon commented 2 weeks ago

Thanks for publishing the new version. I tried 0.0.12 but I'm unfortunately getting the same error, nothing changed 😞

Not a meltano or poetry expert by any means, but shouldn't the faker extra dep be listed here perhaps?

@maxmarcon can you confirm that https://github.com/MeltanoLabs/target-postgres/pull/424 has what you need?

maxmarcon commented 2 weeks ago

@edgarrmondragon I'm confused. I need tap-postgres but you sent me a link to target-postgres ? Thanks

maxmarcon commented 2 weeks ago

This is the one-line change we need to fix the problem:

496

I verified that it works

edgarrmondragon commented 2 weeks ago

@edgarrmondragon I'm confused. I need tap-postgres but you sent me a link to target-postgres ? Thanks

Woops, I got confused 😅. Let me take a look at your PR.