Week 2 feature engineering

[yim-fan]

2nd week feature engineering

• [x] Season
• [x] Hour
• [x] Naive Operation mode
• [x] Direction

[yim-fan]

Have a naive version of operation mode. Will work on a more accurate definition later

[yim-fan]

Have written a function that can convert minute into timestamp and identify hour and season with given starting time as an input variable

[pagand]

When you feel confident about rach module, please go ahead and push to the dev branch.

[pagand]

@yim-fan Back to review: mode: the speed can go up and down abruptly due to noise or sensor error. Please also consider the history, and/or latitude-longitude as well.

considering season to have numeric values is not consistent. currently spring (0) and winter (3) are so far away in the mapping while they are close to each other in a year. A better mapping is one-hot-encoding. such as: spring: [1,0,0,0] summer: [0,1,0,0] fall: [0,0,1,0] winter: [0,0,0,1]

the same issue is for the direction. in addition, the direction can be wrong if the vessel did not get exactly to the H_lat/H_long or N_long/N_lat. This will result in ignoring the vessel change of direction and consider the return direction in the continue of the go direction. take a look at crossing in the vizulization of the main repo in this link to get more insight. I used both the speed in an interval to get the direction.

[yim-fan]

Thanks Pedram, I have my one hot label encoder and standard scaler in the modeling part, which will be available in my next commit. There will also be an update on direction, which will consider the direction in the neighbor records, as long as the speed and water depth,

[yim-fan]

TODO: double check on mode

[yim-fan]

https://github.com/pagand/model_optimze_vessel/blob/6ce51ab10e43f105c94b3d335219dfcff3776477/Feature/feature_engineering.ipynb revision on mode: now uses power, speed through water, and change in STW (because usually in autopilot the vessel should move in constant speed) to classify mode1 and mode2. The classification visualization is also available.

[pagand]

@yim-fan I see a few yellow regions in the middle of the trip. It is possible that in the adversarial trips, the captain see a craft and go to mode 2 in the middle, but it is not the case in normal situations. Can you make sure: 1- those regions are actually correct (they are mode 2 and controlled manually by the captain) 2- if all are correct, all those trips should be labeled as adversarial

[yim-fan]

I have checked with mode 2 in the middle, these are usually caused by sudden change SOG while transmitting. These changes are abnormal, and corresponding trips should be considered as adversarial. I will double check to make sure they are all labeled correctly as adversarial.

[yim-fan]

I made a slight change on adversarial. Now all the mode change happens in the middle should be included in adversarial trips. New code available at: https://github.com/pagand/model_optimze_vessel/blob/e2e15ca9f0b4d50d3bfef7f015694276b22b654c/Feature/feature_engineering.ipynb (scroll down all the way to the end)

[pagand]

Perfect!