search

dlt-hub / verified-sources

Contribute to dlt verified sources 🔥
https://dlthub.com/docs/walkthroughs/add-a-verified-source
Apache License 2.0
72 stars 50 forks source link

feat(mongodb): support mongodb through Arrow #486

Closed IlyaFaer closed 4 months ago

IlyaFaer commented 5 months ago

Towards #438, #484

IlyaFaer commented 5 months ago

@rudolfix, seems to me a small change in the core is needed. There is an error happening:

dlt.extract.exceptions.InvalidStepFunctionArguments: In processing pipe comments: Unable to call IncrementalResourceWrapper: Array type doesn't match type of values set: string vs null. The mapping/filtering function IncrementalResourceWrapper requires first argument to take data item and optional second argument named 'meta', but the signature is (item: Union[Any, List[Any]], meta: Any = None) -> Union[Any, List[Any], NoneType]

I figured out that such a change fixes it:

image

So, if there is nothing in remove_ids, then we don't even do filtering. Otherwise, as far as I can tell, it fails due to empty filtering-out array (which type is not defined, as there is no data in it).