Get One Day Exchange Rates Data

[dan-tang-ssd]

This PR is submitted to fix #124.

Freecurrency API provides exchange rate data up to yesterday, consider time difference and time zones, reserve more buffer to get data of the date before yesterday instead of yesterday.

Command: php artisan app:get-daily-exchange-rates

The schedule job will be executed at 00:21 AM everyday.

---

I intended to modify GetHistoricExchangeRates.php to support this feature. But I realised that this class cannot have another constructor method with different parameters. Finally I copy it as GetOneDayExchangeRates.php.

[dan-tang-ssd]

I have revised program to get yesterday exchange rate data every day at 6:00 AM.

Suppose we should have 1089 (i.e. 33 x 33) exchange_rates records everyday.

I tried several days but the exchange data retrieved is incomplete...

It looks like we only have exchange rate data for the first few base currencies...

[dan-tang-ssd]

The command add jobs to queue normally...

Telescope Jobs tab looks normal...

[dave-mills]

The reason it's only retrieving the first few currencies is because of the rate limiter on the API:

The middleware added to the class (and defined in AppServiceProvider was meant to handle that, by only allowing 5 jobs to complete per minute. I think I messed that up by not specifying a "by" key on the limiter, so it was limiting each individual job to 5 attempts per minute).

I don't quite understand why sometimes it showed all the jobs as "processed" in Telescope, as-in on your version, and sometimes it showed the first few jobs as processed and others as failed.

But, with a few minor tweaks, I think this is now working as intended:

Changes

• added by->($job->date) to the Rate Limiter, meaning it will only run 5 jobs per date, and then all the others will be attempted, get limited, and then get put back on the queue;
• added a retryUntil() method on the job class itself, so it doesn't matter how many tries it takes, it will keep being attempted for 10 minutes before giving up forever. (Previously it would try each job the default number of times, (3) and then end up failing because it was still being rate-limited by that point).

To test locally:

• check sure you job queue is empty (php artisan queue:clear)
• run the command php artisan app:get-one-day-exchange-rates
• startup the job queue worker (php artisan queue:work).
  • It should run the first 5 jobs, then put the rest back onto the queue and wait for 1 minute.
  • Running php artisan queue:monitor default should show you 28 jobs remaining in the queue.
• Leave it running for a few minutes. After 4-5 minutes, all the jobs should have completed, and there should be 1089 entries for "yesterday" in the database.

(It doesn't matter if it's not exactly 1089 - occasionally, the API doesn't seems to have entries for certain combos. I have added a small extra into the ExchangeRateController so that if the rate for a specific date is not found, it will have 1 attempt at retriving the rate from the day before. I think this will be enough to cover most cases of missing data).

@dan-tang-ssd - can you double check this works ok locally? Then I think it's good to go.

[dan-tang-ssd]

Thank you Dave for the enhancement. It works well in my local env.

There is one minor item, in order to put back jobs on the queue, we need to change config item "QUEUE_CONNECTION" from "sync" to "database" in .env file

# QUEUE_CONNECTION=sync
QUEUE_CONNECTION=database

I will update .env file when we have deployment in staging env.

[dan-tang-ssd]

Deployment is completed in staging env.

.env file in staging env is updated for config item "QUEUE_CONNECTION", change from "sync" to "database".

I found that there is no queue created in Forge, I have just created a queue name "default" with connection "database".

I run command to get one day exchange rate data, it works well. Suppose the command will be triggered at 6:00 AM tomorrow by schedule job. I would expect exchange rate data for 2023-08-21 will be existed in database tomorrow morning.

[dan-tang-ssd]

I did a checking in backend database, there is no exchange rate data for 2023-08-21. Not sure whether setup for schedule job is completed in staging env. I will check and feedback.

[dan-tang-ssd]

Checked that scheduler has not been created for staging env before. I have just created a scheduler for staging env. Suppose the schedule job for getting exchange rate data will be executed at 6:00 AM tomorrow. It is expected to have 2023-08-22 exchange rate data when I do checking tomorrow morning.

I have just manually run the command for getting exchange rate data for 2023-08-21. It manages to complete within a few minutes, exchange rate data for 2023-08-21 is successfully retrieved.

[dan-tang-ssd]

I did a checking in backend database this morning, there is no exchange rate data for 2023-08-22.

@dave-mills , I have created a queue on 21 Aug and a scheduler on 22 Aug in staging env. Is there anything I missed in setup...?

[dave-mills]

Looking at the Forge scheduler output, I see it says something about not recognising the value "Sunday".

Turns out when specifying a day of the week we should use number values instead of the strings. I've sent a pull request to fix this. #218

[dan-tang-ssd]

We have deployed a fix for schedule job on 2023-08-23. I did a checking on scheduler yesterday, no error message. I suppose the scheduler is working properly.

We had exchange rate data for 2023-08-20 and 2023-08-21. The scheduler should run schedule job on 2023-08-24 6:00 AM to get exchange rate data for 2023-08-23.

Just did a checking in front end with positive result. Different exchange rate data showed for 2023-08-20, 2023-08-21 and 2023-08-23.

I would consider this PR can be closed now.

---

Screen shots: