data-solution-automation-engine / data-warehouse-automation-metadata-schema

Generic interface exchange format for Data Warehouse Automation and ETL generation.
GNU Lesser General Public License v3.0
38 stars 11 forks source link

What about Meta Data from previously unknown header info #3

Closed papyr closed 4 years ago

papyr commented 4 years ago

in my case, the source CSV data file and header content/types change often -- basically unknown, while parsing or before parsing I need to first Discover the Data Types of the data in the cols.

I just need to load & save the data into a Sql table based on whatever data & DataTypes there are in the CSV file. Can I auto load the CSV data and datatypes into the DB automatically with ETL box. I saw your expando example but was a little lost how to achieve a high level task

For e.g high level algorithm, how would I achieve this in your lib.

// Scan first 9 rows to discover the datatypes in the CSV file cols
var etlPrasedFile = Data_WareHOuse.Open/parseCSV(someUnknownData.csv).ScanRows(9)
foreach (var col in etlPrasedFile )
new dataTable.addColumn(etlParsedFile.GetNextHeaderCol)
CreateNewSqlServerTable
LoadDatatoSqlTable??

thanks

RoelantVos commented 4 years ago

Hi – I know what you mean, in the past I have used C# to read the columns and create a table in a database to be able to stash the CSV somewhere. The library is intended to capture source-to-target mappings more geared towards transformations between one database table and another.

Having said that, you can map this from a file to a table also and store this but in this case the structure must be known.

From: papyr notifications@github.com Sent: Sunday, 16 February 2020 17:35 To: RoelantVos/Data_Warehouse_Automation_Metadata_Interface Data_Warehouse_Automation_Metadata_Interface@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [RoelantVos/Data_Warehouse_Automation_Metadata_Interface] What about Meta Data from previously unknown header info (#3)

in my case, the source CSV data file and header content/types change often -- basically unknown, while parsing or before parsing I need to first Discover the Data Types of the data in the cols.

I just need to load & save the data into a Sql table based on whatever data & DataTypes there are in the CSV file. Can I auto load the CSV data and datatypes into the DB automatically with ETL box. I saw your expando example but was a little lost how to achieve a high level task

For e.g high level algorithm, how would I achieve this in your lib.

// Scan first 9 rows to discover the datatypes in the CSV file cols var etlPrasedFile = Data_WareHOuse.Open/parseCSV(someUnknownData.csv).ScanRows(9) foreach (var col in etlPrasedFile ) new dataTable.addColumn(etlParsedFile.GetNextHeaderCol) CreateNewSqlServerTable LoadDatatoSqlTable??

thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RoelantVos/Data_Warehouse_Automation_Metadata_Interface/issues/3?email_source=notifications&email_token=AHRXHVJ3V2YU2KHQEITA2QLRDDUANA5CNFSM4KWARGD2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IN2NK5Q , or unsubscribe https://github.com/notifications/unsubscribe-auth/AHRXHVKO46YVE2JST7N5PKLRDDUANANCNFSM4KWARGDQ . https://github.com/notifications/beacon/AHRXHVP5UY2BRM3ESELNXSLRDDUANA5CNFSM4KWARGD2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IN2NK5Q.gif

papyr commented 4 years ago

Well, then I can still use it for another use case 👍

How do I go about mapping one table to another table in a different database. Do you have a sample posted online/wiki

RoelantVos commented 4 years ago

Hi there!

There are a few samples in the example project, one of which has connectivity information included also.

It’s basically an object mapped to another object, with each object having a connection you can specify.

Kind regards,

Roelant

From: papyr notifications@github.com Sent: Sunday, 15 March 2020 21:12 To: RoelantVos/Data_Warehouse_Automation_Metadata_Interface Data_Warehouse_Automation_Metadata_Interface@noreply.github.com Cc: Roelant Vos roelant@ravos.nl; Comment comment@noreply.github.com Subject: Re: [RoelantVos/Data_Warehouse_Automation_Metadata_Interface] What about Meta Data from previously unknown header info (#3)

Well, then I can still use it for another use case 👍

How do I go about mapping one table to another table in a different database. Do you have a sample posted online/wiki

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RoelantVos/Data_Warehouse_Automation_Metadata_Interface/issues/3#issuecomment-599196168 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AHRXHVLXN25CPIUULDT2EWDRHSZWNANCNFSM4KWARGDQ . https://github.com/notifications/beacon/AHRXHVIJWWQOVDTI7GN2ALTRHSZWNA5CNFSM4KWARGD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEO3QECA.gif

RoelantVos commented 3 years ago

From: Roelant Vos roelant@ravos.nl Sent: Sunday, 28 February 2021 18:38 To: 'Fodi Dervidis' fodi@outlook.com; 'Sam Spencer' SSpencer@allianz-assistance.com.au; 'aaronblakeney@yahoo.com.au' aaronblakeney@yahoo.com.au; 'Allan Duckwitz' ADuckwitz@allianz-assistance.com.au; 'Adam Raffa' ARaffa@allianz-assistance.com.au; 'devereaux64@gmail.com' devereaux64@gmail.com; 'hellobenchen@gmail.com' hellobenchen@gmail.com; 'lmweber@hotmail.com' lmweber@hotmail.com; 'luciano.bastian@gmail.com' luciano.bastian@gmail.com; 'mehdireleh@yahoo.com' mehdireleh@yahoo.com; 'mwaters1973@gmail.com' mwaters1973@gmail.com; 'Ashley Peet' APeet@allianz-assistance.com.au; 'ej@live.nl' ej@live.nl Subject: RE: [EXT] LAN party - March 12th

Thanks Fodi, looks like we’re good with the tables then!

Any ideas on games and/or setup?

From: Fodi Dervidis <fodi@outlook.com mailto:fodi@outlook.com > Sent: Sunday, 28 February 2021 08:43 To: 'Sam Spencer' <SSpencer@allianz-assistance.com.au mailto:SSpencer@allianz-assistance.com.au >; aaronblakeney@yahoo.com.au mailto:aaronblakeney@yahoo.com.au ; 'Allan Duckwitz' <ADuckwitz@allianz-assistance.com.au mailto:ADuckwitz@allianz-assistance.com.au >; 'Adam Raffa' <ARaffa@allianz-assistance.com.au mailto:ARaffa@allianz-assistance.com.au >; benwiedmail@gmail.com mailto:benwiedmail@gmail.com ; devereaux64@gmail.com mailto:devereaux64@gmail.com ; hellobenchen@gmail.com mailto:hellobenchen@gmail.com ; lmweber@hotmail.com mailto:lmweber@hotmail.com ; luciano.bastian@gmail.com mailto:luciano.bastian@gmail.com ; mehdireleh@yahoo.com mailto:mehdireleh@yahoo.com ; mwaters1973@gmail.com mailto:mwaters1973@gmail.com ; 'Ashley Peet' <APeet@allianz-assistance.com.au mailto:APeet@allianz-assistance.com.au >; ej@live.nl mailto:ej@live.nl ; 'Scott Diprose' <SDiprose@allianz-assistance.com.au mailto:SDiprose@allianz-assistance.com.au >; Roelant Vos <roelant@ravos.nl mailto:roelant@ravos.nl > Subject: Re: [EXT] LAN party - March 12th

I have a folding trestle table. Should be ok for another 2-4 people. Also have some stacking chairs. I could drop them off beforehand.

Get Outlook for Android https://aka.ms/ghei36


From: Roelant Vos <roelant@ravos.nl mailto:roelant@ravos.nl > Sent: Friday, February 26, 2021 4:15:29 PM To: 'Sam Spencer' <SSpencer@allianz-assistance.com.au mailto:SSpencer@allianz-assistance.com.au >; aaronblakeney@yahoo.com.au mailto:aaronblakeney@yahoo.com.au <aaronblakeney@yahoo.com.au mailto:aaronblakeney@yahoo.com.au >; 'Allan Duckwitz' <ADuckwitz@allianz-assistance.com.au mailto:ADuckwitz@allianz-assistance.com.au >; 'Adam Raffa' <ARaffa@allianz-assistance.com.au mailto:ARaffa@allianz-assistance.com.au >; benwiedmail@gmail.com mailto:benwiedmail@gmail.com <benwiedmail@gmail.com mailto:benwiedmail@gmail.com >; devereaux64@gmail.com mailto:devereaux64@gmail.com <devereaux64@gmail.com mailto:devereaux64@gmail.com >; fodi@outlook.com mailto:fodi@outlook.com <fodi@outlook.com mailto:fodi@outlook.com >; hellobenchen@gmail.com mailto:hellobenchen@gmail.com <hellobenchen@gmail.com mailto:hellobenchen@gmail.com >; lmweber@hotmail.com mailto:lmweber@hotmail.com <lmweber@hotmail.com mailto:lmweber@hotmail.com >; luciano.bastian@gmail.com mailto:luciano.bastian@gmail.com <luciano.bastian@gmail.com mailto:luciano.bastian@gmail.com >; mehdireleh@yahoo.com mailto:mehdireleh@yahoo.com <mehdireleh@yahoo.com mailto:mehdireleh@yahoo.com >; mwaters1973@gmail.com mailto:mwaters1973@gmail.com <mwaters1973@gmail.com mailto:mwaters1973@gmail.com >; 'Ashley Peet' <APeet@allianz-assistance.com.au mailto:APeet@allianz-assistance.com.au >; ej@live.nl mailto:ej@live.nl <ej@live.nl mailto:ej@live.nl >; 'Scott Diprose' <SDiprose@allianz-assistance.com.au mailto:SDiprose@allianz-assistance.com.au > Subject: RE: [EXT] LAN party - March 12th

That will house 2-3 gaming enthusiasts 😊.

I only have the dining table which would house ~5 probably and a camping table for another 2-3.

From: Sam Spencer <SSpencer@allianz-assistance.com.au mailto:SSpencer@allianz-assistance.com.au > Sent: Friday, 26 February 2021 15:34 To: 'roelant@ravos.nl' <roelant@ravos.nl mailto:roelant@ravos.nl >; aaronblakeney@yahoo.com.au mailto:aaronblakeney@yahoo.com.au ; Allan Duckwitz <ADuckwitz@allianz-assistance.com.au mailto:ADuckwitz@allianz-assistance.com.au >; Adam Raffa <ARaffa@allianz-assistance.com.au mailto:ARaffa@allianz-assistance.com.au >; benwiedmail@gmail.com mailto:benwiedmail@gmail.com ; devereaux64@gmail.com mailto:devereaux64@gmail.com ; fodi@outlook.com mailto:fodi@outlook.com ; hellobenchen@gmail.com mailto:hellobenchen@gmail.com ; lmweber@hotmail.com mailto:lmweber@hotmail.com ; luciano.bastian@gmail.com mailto:luciano.bastian@gmail.com ; mehdireleh@yahoo.com mailto:mehdireleh@yahoo.com ; mwaters1973@gmail.com mailto:mwaters1973@gmail.com ; Ashley Peet <APeet@allianz-assistance.com.au mailto:APeet@allianz-assistance.com.au >; ej@live.nl mailto:ej@live.nl ; Scott Diprose <SDiprose@allianz-assistance.com.au mailto:SDiprose@allianz-assistance.com.au > Subject: RE: [EXT] LAN party - March 12th

Hey hey,

I have an (approx.) 6x2.5ft folding table I can bring along. No chairs though.

UT2004, BF1, or RS6 are my preferences. But I’m happy to be cannon fodder on Supreme Commander if it comes to it.

Regards,

Sam Spencer BI Analyst / Developer mailto:SSpencer@allianz-assistance.com.au SSpencer@allianz-assistance.com.au

310 Ann St, Brisbane QLD 4000 PO BOX 162, Toowong 4066 https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.allianzpartners.com.au%2F&data=04%7C01%7C%7Ce418e4ac48364be5619008d8da1df1e6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637499169440579274%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h4QFPKjm3mF5n6LZfj7glbbhHefyr0h3cLdG9IqhPFk%3D&reserved=0 www.allianzpartners.com.au

https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2FAllianzAssistanceAustralia&data=04%7C01%7C%7Ce418e4ac48364be5619008d8da1df1e6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637499169440589269%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JvXCaQoiNtTzZbvhYhc0OYo2vJL3swEPNeaJSh6EsUc%3D&reserved=0

https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fallianzassist&data=04%7C01%7C%7Ce418e4ac48364be5619008d8da1df1e6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637499169440599258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iFzNM6bE%2BTMd4ISVSKUzNtMjCnBBGeL0TO7FMca89vY%3D&reserved=0

https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fallianz-global-assistance-australia&data=04%7C01%7C%7Ce418e4ac48364be5619008d8da1df1e6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637499169440599258%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cVTl2tViFae3nncI0Vq17uIM7DKGClyy1hHCSxlLTkA%3D&reserved=0

https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplus.google.com%2F116703753643833779792%2Fposts&data=04%7C01%7C%7Ce418e4ac48364be5619008d8da1df1e6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637499169440609252%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yPhsGaUOmGa8bD4Lb4RAfNdb8KyLvHBc3tH%2FCR7IkUQ%3D&reserved=0

From: roelant@ravos.nl mailto:roelant@ravos.nl <roelant@ravos.nl mailto:roelant@ravos.nl > Sent: Friday, 26 February 2021 2:30 PM To: aaronblakeney@yahoo.com.au mailto:aaronblakeney@yahoo.com.au ; Allan Duckwitz <ADuckwitz@allianz-assistance.com.au mailto:ADuckwitz@allianz-assistance.com.au >; Adam Raffa <ARaffa@allianz-assistance.com.au mailto:ARaffa@allianz-assistance.com.au >; benwiedmail@gmail.com mailto:benwiedmail@gmail.com ; devereaux64@gmail.com mailto:devereaux64@gmail.com ; fodi@outlook.com mailto:fodi@outlook.com ; hellobenchen@gmail.com mailto:hellobenchen@gmail.com ; lmweber@hotmail.com mailto:lmweber@hotmail.com ; luciano.bastian@gmail.com mailto:luciano.bastian@gmail.com ; mehdireleh@yahoo.com mailto:mehdireleh@yahoo.com ; mwaters1973@gmail.com mailto:mwaters1973@gmail.com ; Sam Spencer <SSpencer@allianz-assistance.com.au mailto:SSpencer@allianz-assistance.com.au >; Ashley Peet <APeet@allianz-assistance.com.au mailto:APeet@allianz-assistance.com.au >; ej@live.nl mailto:ej@live.nl ; Scott Diprose <SDiprose@allianz-assistance.com.au mailto:SDiprose@allianz-assistance.com.au > Subject: [EXT] LAN party - March 12th

Hey everyone,

Since you’re interested in having a LAN, let’s talk games!

Some ideas;

I’ll organise food & drinks, but what would be really helpful is to bring 2 or so camping tables and chairs. Since we almost-moved-but-didn’t we sold and gave away lots of stuff so we’re down on tables 😊. Also necessary are power blocks and extension cord, we need a few of those.

Kind regards,

Roelant

Note: This email, including any attachments, has been sent by AWP Australia Pty Ltd (ABN 52 097 227 177) trading as Allianz Global Assistance and is intended solely for the addressee. It is confidential, may contain personal information and may be subject to legal professional privilege. If you have received this by mistake, confidentiality and any legal privilege are not waived or lost and we ask that you contact the sender and delete and destroy this and any other copies. In relation to any lawful use you may make of the contents of this email, you must ensure that you comply with the Privacy Act (C'th) 1988 and you should note that the contents may be subject to copyright and therefore may not be reproduced, communicated or adapted without the express consent of the owner of the copyright. Allianz Global Assistance will not be liable in connection with any data corruption, delay, computer virus or unauthorised access, interception, or amendment to the contents of this email. If you no longer want to receive messages from us, please contact us at Allianz Global Assistance, PO Box 162 Toowong, Qld, 4066, ph +61 7 3305 7000.