search

Alation / Allie-SDK

A Python library for the Alation REST APIs.
https://alation.github.io/Allie-SDK/
Apache License 2.0
9 stars 2 forks source link

allie_sdk\methods\virtual_datasource.py passed wrong type to async_post #22

Closed laskmat closed 3 months ago

laskmat commented 3 months ago

hey, this looks like a simple bug

Method post_metadata from allie_sdk\methods\virtual_datasource.py

this part

`....

    payload_d = [item.generate_api_post_payload() for item in vds_objects]

    # add line feeds between json payload dicts for jsonl format

    payload_jsonl = '\n'.join(json.dumps(p) for p in payload_d)

    LOGGER.debug(payload_jsonl)    

    async_results = self.async_post(f'{self._vds_endpoint}{ds_id}', payload=payload_d, query_params=params)

.... `

says that payload for POST should be str. BUT async_post contains chunking data object list

... batches = self._batch_objects(payload, batch_size) ... which means that we're getting chunked str data instead of "list of jsons" when having passed wrong param at post_metadata. Impact for scripting with using this is small (relatively) because for now last item from payload list is always damaged. I'll rise PR for fix with that (also have to check other cases where data chunks are used)

@diethard-at-alation is it worth to mark this as bug per se?

laskmat commented 3 months ago

hmm not very clear here - issue exists only when having custom_fields on board.

ignaszewskir commented 3 months ago

Hey laskmat. Thanks for using the AllieSDK! What exactly is the issue? The code you attached should be async_results = self.async_post(f'{self._vds_endpoint}{ds_id}', payload=payload_jsonl, query_params=params) Could you send please the sample payload please and the client code?

Radek

diethard-at-alation commented 3 months ago

We looked into this a bit closer and noticed the issue as well. We will make some changes and publish them once tested.

ignaszewskir commented 3 months ago

Hello @laskmat. I have fixed the problem where string data payloads where erroneusly batched. The API endpoints for the virtual data source and virtual file source expect a data payload of type string and the sequence of objects appearing in the payload is important. For example if you specify 'remove_not_seen' parameter, then the objects that are not part of payload would be deleted. I added a new, internal api to handle those types of payloads that do not require internal batching. Thank you for testing and reporting your issues!

ignaszewskir commented 3 months ago

Improvements implemented in the next pull request. Closing.

laskmat commented 3 months ago

hello @ignaszewskir I'm really sorry - had holiday and didnt notice message here! I'm happy to hear that @diethard-at-alation. thanks guys! yep, I'm doing some automatization with this lib so I guess I could find more stuff. regards