META: Find optimal params for preprocessing

[lucasfbn]

Find optimal parameters for preprocessing, #106

• Objectives

  • Find
  • [x] ticker_min_len
    • This setting influences two behaviour:
      • When ticker will make it into the dataset (min number of sentiment entries)
      • When a time series is tradeable (if we set it to 6 then the 5 previous sentiment entries are being ignored during sequence generation)
    • Therefore, we set it to 2 to make sure:
      • We use as many ticker as possible for training
      • We make each ticker tradeable as early as possible from the first mention within the sentiment data
  • [x] sequence_len
    • Set to 14 since most optimal offsets are between 7 and 14.
  • [x] min_sequence_len
    • Remains at 2, we still need at least two sequences to train a ticker.
  • [x] Find optimal setting for scale_sentiment_data_daywise (e.g. whether of not to scale the sentiment data daywise)
    • Doesn't make a difference correlation and offset wise. We will keep it on False for now.
  • Abbreviated from above-mentioned
  • [x] price_data_start_offset
    • Set to sequence_len + 1.

• Steps

  • [x] Make dashboard

  • [x] Plot timeseries

  • [x] Plot time lagged cross correlation

  • [x] Plot max tlcc of common combinations

  • [x] Plot bulk data statistics

  • [x] Make sure interpretation is correct

    Yes. Can be tracked by shifting a dataframe:

    df = pd.Series(range(10))
    df = df.shift(-2)

    The resulting dataframe will be shifted "backwards". E.g. the cross correlation will also be calculated with the backwards shifted y timeseries.

  • [x] Make new dataset without that many constraints (e.g. lower ticker_min_len)

  • [x] Results

    scale_sentiment_data_daywise=False Correlation Offset

    scale_sentiment_data_daywise=True Basically same as above.