search

Daimler666 / Data-Lake-Ticket-Platform

1 stars 0 forks source link

AQUA MASTER DATA CHECK NEED #5

Open Daimler666 opened 7 years ago

Daimler666 commented 7 years ago

/Raised by Felix Liu/

Status:

  1. New version of AQUA master data has acquired from source team;
  2. Data has already been loaded into INT Edgenode;

Task:

  1. The validation of data (only one csv file) should be checked
Daimler666 commented 7 years ago

/Replied by Anil/

Issues:

  1. The header is not maintained in the master data file, so cannot identify fields and values;
  2. Two field values are missed and not be maintained according to data modeling sheet;
  3. Check which delimiter AQUA files should be used to maintain later, comma(,) or semi-colon(;).
Daimler666 commented 7 years ago

/Replied by Debdatta/

Status:

  1. Supposed to find the connection details from AQUA;

Confusion:

  1. Why source team send us csv files?
Daimler666 commented 7 years ago

/Replied by Felix Liu/

Status:

  1. The headers missed have been added;
  2. Export SQL and format descrption have been provided by source team;
  3. Have gotten the user of AQUA and is working on research the source data structure;
  4. Have go through this version of data one time and it looks fine.

Task:

  1. Export the data ourselves;
  2. Check the data format is the same or not;
  3. This version of data should be double check.

Suggestion:

  1. It seems csv is the only possible data carrier;
  2. We still cannot directly connect to Teradata database but we can define export format and control export procedure;
  3. You guys can adjust the data lake table according to raw data directly.
Daimler666 commented 7 years ago

/Replied by Debdatta/

Status:

  1. We are taking AQUA master data update as a lower priority than DMS.

Suggestion:

  1. It is better to have quick phone call about AQUA on monday. (to Felix Liu)
Daimler666 commented 7 years ago

/Replied by Felix Liu/

Issues Summary: AQUA data still need to be processed and populated.

Daimler666 commented 7 years ago

/Replied by Anil/

Question:(to Felix Liu)

  1. To maintain MasterData datalake table with only 52 fields as per the header in the sample excel sheet and load it right ?
Daimler666 commented 7 years ago

/Replied by Felix/

Status:

  1. Please do that ASAP.
Daimler666 commented 7 years ago

/Replied by Anil/

Status:

  1. “date_of_credit” field is not extracted in this master data, it is one of the field in the composite primary key that we need to consider in the logic.
  2. So we can’t load this data, kindly extract the file with including “date_of_credit” field.
Daimler666 commented 7 years ago

/Replied by Felix Liu/

Status:

  1. We found one key column (“date_of_credit” ) is missing again in the latest version of AQUA data.

Suggestion:

  1. We can login the Microstrategy then draft the sql and export the data ourselves.
Daimler666 commented 7 years ago

/Replied by Madhu/

Status: 1.Pending at Source side.

Daimler666 commented 7 years ago

/Replied by Felxi/

Status:

  1. we have received and checked the sample data, I put them in below path: /data/landing_zone/BMBS/AQUA/data/AQUA_Sample_11082017
  2. For the next step:
    • Creation of the historical data files, starting and planned for 30.08. until 01.09.2017 => AQUA Supplier
    • Automatic transition for regular data transfer (weekly on Tuesday morning CET) starting 05.09.2017 => AQUA Supplier