Fix timeout issue on data inserting script

Reviewer: @suzinyou Estimate: 20mins

Ticket

Fixes: [FIX]

Description

Sometimes the scripts fails due to a timeout error (or when the internet connection is unstable). This script fixes this by retrying twice with an exponential wait time between each try and then returning None if after trying twice there is a failure.

Goal

Changes

Future Tasks (optional)

How has this been tested?

To-do before merge (optional)

Checklist

Fill with x for completed.

[x] My code follows the style guidelines of this project
[x] I have reviewed my own code to ensure good quality
[x] I have tested the functionality of my code to ensure it works as intended
[x] I have resolved merge conflicts

(Delete any items below that are not relevant)

[ ] I have updated the automated tests
[ ] I have updated the scripts in scripts/
[ ] I have updated the requirements
[ ] I have updated the README file
[ ] I have updated affected documentation
[ ] I have added a blogpost in Latest Updates
[ ] I have updated the CI/CD scripts in .github/workflows/
[ ] I have updated the Terraform code

I am getting this error when generating feedback..

...
Response Feedback successfully added for query_id: 10012.0
Response Feedback successfully added for query_id: 10011.0
Response Feedback successfully added for query_id: 10013.0
Processing content feedback...
Processing urgency detection...
Urgency Detection successfully processed
Traceback (most recent call last):
  File "/Users/suzinyou/IDinsight/aaq-core/core_backend/add_new_data_to_db.py", line 444, in <module>
    random_dates = [
  File "/Users/suzinyou/IDinsight/aaq-core/core_backend/add_new_data_to_db.py", line 445, in <listcomp>
    create_random_datetime_from_string(start_date) for _ in range(len(df))
  File "/Users/suzinyou/IDinsight/aaq-core/core_backend/add_new_data_to_db.py", line 294, in create_random_datetime_from_string
    random_number_of_days = random.randint(0, time_difference.days)
  File "/Users/suzinyou/anaconda3/envs/aaq/lib/python3.10/random.py", line 370, in randint
    return self.randrange(a, b+1)
  File "/Users/suzinyou/anaconda3/envs/aaq/lib/python3.10/random.py", line 353, in randrange
    raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0, -1090, -1090)

Any idea why? I used --start-date 23-08-27 in the command 🤔

[EDIT] I used the wrong order.. it should have been 27-08-23 🤦 !!

I purged the DB volume (of all 13k rows 😭 ) and reran the script with pd.read_csv(..., nrows=10), and everything ran successfully.

Can we add a check for --start-date to be always a past date?

Yes, my bad, I am used to the DD-MM-YY format. But I will add the check

IDinsight / ask-a-question