datacleaner / DataCleaner

The premier open source Data Quality solution
GNU Lesser General Public License v3.0
595 stars 180 forks source link

Is it possible to use DataCleaner to call an OData API and receive the response in JSON or XML format, and use this response's structure and data as the Source/Datastore? #1938

Closed ruben-jardim closed 1 year ago

ruben-jardim commented 1 year ago

Bottom line: I'm trying to make DataCleaner connect to an OData API to receive a JSON or XML response and use it as data source => without having to use Postman to make the call, then download the JSON/FILE file only to then finally register and use it as a data source (long story short: there is a reason for having this direct call necessity).

I couldn't figure it out by myself after playing around with DataCleaner. Maybe I missed something, or maybe the feature isn't there either. The closest I got was trying to register it as a JSON url, but it didn't allow me perform basic authentication. Either way I thought it'd be worth checking with you.

In other words: I'd like to know if it's possible to register a Datastore that, much like Postman, performs:

  1. a GET call to an OData REST API by...
  2. ...using Basic Authentication -> username and password
  3. then receives the JSON or XML response (most likely storing the response in an in-memory table)
  4. interprets its structure and data
  5. ...allow me to use it as a Datastore source

Example: DataCleaner (like Postman) performs a GET request to

https:///odata/v2/User?$format=JSON

and receives a response in JSON format with a list of all existing Users in that platform's database with all their columns like "userId", "lastName", "gender", etc.

Please let me know if it's possible, or if it requires coding this as a new feature. Thanks in advance for your support.

kaspersorensen commented 1 year ago

Hi @ruben-jardim It's not possible out of the box, no. I guess this would have to be implemented as a new type of datastore, or as an addition to the JSON datastore's capabilities.

ruben-jardim commented 1 year ago

Thanks for letting me know. Then we can close this topic with the recomendation to add this feature at your earliest convenience. If I can come around to try and do it myself, I'll let you know.