search

OHDSI / WhiteRabbit

WhiteRabbit is a small application that can be used to analyse the structure and contents of a database as preparation for designing an ETL. It comes with RabbitInAHat, an application for interactive design of an ETL to the OMOP Common Data Model with the help of the the scan report generated by White Rabbit.
http://ohdsi.github.io/WhiteRabbit
Apache License 2.0
174 stars 85 forks source link

Ability to upload an existing mapping plan in CSV #308

Open mpatel-cai opened 2 years ago

mpatel-cai commented 2 years ago

Is your feature request related to a problem? Please describe. I may be mistaken, but Rabbit In A Hat seems like it would be most valuable when it is used to design an ETL from scratch. However, a lot of the times, an organization is in the middle of an ETL and corrections need to be made to the mappings. This would be a great platform to build out a use case where a csv of the existing mappings between source and target table-fields can also be uploaded after the scan report for the source and the data model csv for the target.

Describe the solution you'd like Like the ability to import a scan report and a custom cdm as a csv, I would also like the ability to import a csv of the field-to-field mappings. A similar mapping file with additional comments could then be exported as a csv and a diagram to share with engineering teams, for example.

Describe alternatives you've considered I have struggled with finding such a tool and Rabbit In A Hat is the only one that has come close.

MaximMoinat commented 2 years ago

Thanks for your suggestion. I can see that being a nice and useful addition. Could I ask you to share what data format you are thinking of, to import into RiaH?

mpatel-cai commented 2 years ago

@MaximMoinat I am thinking a csv that contains the following fields:

  1. Source Table
  2. Source Field
  3. Target Table
  4. Target Field

I think in tandem, the ability to export a clean csv of the same while designing the ETL would be fantastic as well.

From: Maxim Moinat @.> Reply-To: OHDSI/WhiteRabbit @.> Date: Tuesday, September 28, 2021 at 12:25 PM To: OHDSI/WhiteRabbit @.> Cc: Meera Patel @.>, Author @.***> Subject: Re: [OHDSI/WhiteRabbit] Ability to upload an existing mapping plan in CSV (#308)

Thanks for your suggestion. I can see that being a nice addition. Could I ask you to share what data format you are thinking of, to import into RiaH?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FNzkzCzpyjOh6g40T40_gL%3Fdomain%3Dgithub.com&data=04%7C01%7Cmpatel%40concertai.com%7Cb4e3edbc1832482148a108d9829ca514%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637684431593137482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DeFBIUlYiWpEj65MSBCgNa5e6SY0pkmICKaT%2F4tBvwQ%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FWusSCADgkZs46Evc8RSfC%3Fdomain%3Dgithub.com&data=04%7C01%7Cmpatel%40concertai.com%7Cb4e3edbc1832482148a108d9829ca514%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637684431593137482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=45HIgzwGh%2F9PRK186ho0%2F0LGxFbyRRjLmt5X9TsvemM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2F0gbdCBBj0ZIj68gFNOdXj%3Fdomain%3Dapps.apple.com&data=04%7C01%7Cmpatel%40concertai.com%7Cb4e3edbc1832482148a108d9829ca514%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637684431593147481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2Fn%2B9l6YiZrVSYTVmdqg0cWguCCDV0xA9cm%2FeMV2faK4%3D&reserved=0 or Androidhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FELFHCDklnjTqv3RCZkkfy%3Fdomain%3Dplay.google.com&data=04%7C01%7Cmpatel%40concertai.com%7Cb4e3edbc1832482148a108d9829ca514%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637684431593147481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AHLVUXrPkJrxPDdn0DgIyQyeIheK4YFxncTM6ZXG9jM%3D&reserved=0.

MaximMoinat commented 2 years ago

Thanks Meera.

For the Export of the mappings, the most recent RiaH version has the following export options that might be useful:

image

mpatel-cai commented 2 years ago

@Maxim @.***>

Yes, but it is just a independent list of source and target and the Table Mapping is a table-to-table mapping but not specific to field.

From: Maxim Moinat @.> Reply-To: OHDSI/WhiteRabbit @.> Date: Monday, October 4, 2021 at 1:02 AM To: OHDSI/WhiteRabbit @.> Cc: Meera Patel @.>, Author @.***> Subject: Re: [OHDSI/WhiteRabbit] Ability to upload an existing mapping plan in CSV (#308)

Thanks Meera.

For the Export of the mappings, the most recent RiaH version has the following export options that might be useful:

[Image removed by sender. image]https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FpquHC31PD8hqnWEIgn1O0%3Fdomain%3Duser-images.githubusercontent.com&data=04%7C01%7Cmpatel%40concertai.com%7C4a74f22253df477eb7a108d9870d5e6a%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637689313757762241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ggvWr7bxlmX5lIYHQ%2BUqEwFaO34Rs8yzJ%2Bln2UrWnuo%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2F4fktC4xP0ZTgXRvux_vl2%3Fdomain%3Dgithub.com&data=04%7C01%7Cmpatel%40concertai.com%7C4a74f22253df477eb7a108d9870d5e6a%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637689313757762241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=391nWgYlGJsZiP5Wz4%2B5m81cNzNH0Q8XbgyNHzdlwbY%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2F5CciC5yXGZTEBwDuyor6l%3Fdomain%3Dgithub.com&data=04%7C01%7Cmpatel%40concertai.com%7C4a74f22253df477eb7a108d9870d5e6a%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637689313757772231%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DiMG3HnJItyhmKnCoiveN5P8SlSGuDdJnmdYxII%2FDVI%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FoY_TC684J1hz5ZBi5FB4I%3Fdomain%3Dapps.apple.com&data=04%7C01%7Cmpatel%40concertai.com%7C4a74f22253df477eb7a108d9870d5e6a%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637689313757782226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HhHBwo6hwmldIHi0XnCDNuwG64pQhOEiwQ%2FyYrQNnxA%3D&reserved=0 or Androidhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fprotect-us.mimecast.com%2Fs%2FlcS4C732KZflDnJTqAYax%3Fdomain%3Dplay.google.com&data=04%7C01%7Cmpatel%40concertai.com%7C4a74f22253df477eb7a108d9870d5e6a%7C61a32d1a5fde47799b503fc4a615ddc4%7C1%7C0%7C637689313757782226%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=t6vzc1Vg7Xk0kyeUEeYZ%2BVJRh4z0VBEeylWgmggDBBU%3D&reserved=0.