summarizing findings for cleanup and observations of data.

[chriskuz]

Going to be adding highlights to this thread periodically for tracking better some of the work that went into understanding the data.

[chriskuz]

Nulls:

• JetBlue data subset from kaggle itineraries data showcased a null percentage of 0.82% out of the total elements in the dataframe. This is different from saying that 1.86% of the rows are null....it's total elements.
• the offending columns of nulls include: totalTravelDistance (56,436 elements), segmentsEquipmentDescription (52,443 elements), segmentsDistance (19,469 elements)
• when digging deeper into this and observing pairings we found 6,715,947 rows of the above 3 columns did not contain null (this is a good thing as we are able to salvage much with this information.
• when understanding the groupings of columns null existence, we found 108,493 rows in total will be affected by some existence of a null value
• the resolution was to drop nulls entirely.
• the resulting dataframe was left with 6,715,947 rows where 0.01% of our rows were to be removed. A total of 108,493 rows were removed.

[chriskuz]

Error labeling segmentsEquipmentDescription:

• segmentsEquipmentDescription is a column that showcases what kind of aircraft was flown for a route
• the column is inclusive of "trip legs" where equipment is delimited by a || symbol
• the column contains a large amount of inconsistencies (including the overuse of the || delimiter; a lack of information for leg routes; inconsistency with similar aircraft naming; and blatant fake aircraft generation).
• NO RESOLUTION FOUND

There exists an idea where a more consistent column that correctly uses a || delimiter could help us fill in the gaps on any missing information for this. However, this would likely mean we would need smart use of regex which is really annoying.

[chriskuz]

Duplicated Features (segmentsAirlineCode, segmentsAirlineName):

• the segmentsAirlineCode column as well as the segmentsAirlineName column is clean and consistent enough to be considered as a helper column for the cleanup of segmentsEquipmentDescription.
• these features are effectively duplicates of one anothere where the segmentsAirlineName column is a full name and the segmentsAirlineCode column is the abbreviated prefixed callsign of the airline.
• JetBlue partners with many different airlines for their routes (source). As a result, specific partners are observed here through the varying airline names and the varying airline codes.
• the largest "pure JetBlue" routes we found (single leg to triple leg) summed up to 6,708,602 rows. The amount of impure routes summed up to 7,345 rows.
• NO RESOLUTION FOUND

It is likely best for us to consider the removal of the impure routes which will shorten the table a little bit. Also, since this will create pure JetBlue routes, we can consider the utter removal of this column as well as the airline code column. The filter must happen first based on these helper columns before a consideration of removing the columns. There might be a case to keep at least one of these columns to take advantage of the delimiters which may help a model understand the price correlation with multi leg routes. No resolution found for now.