`kyna-data-import` fails with the "API Limit Reached" exception - but there are still credits remaining.

[vicsharp-shibusa]

This is some sort of defect in the accounting of credits used.

Splits for OFIX.US
Dividends for MXI.US
EOD Prices for FFBC.US
Kyna.Infrastructure.DataImport.ApiLimitReachedException: Available credits (-52) is less than expected cost of 1 (InvokeEodCallsForExchangeAsync)
   at Kyna.Infrastructure.DataImport.EodHdImporter.CheckApiLimit(Nullable`1 expectedCost, String caller) in C:\repos\kyna\dotnet\libs\Kyna.Infrastructure\DataImport\EodHdImporter.cs:line 989
   at Kyna.Infrastructure.DataImport.EodHdImporter.<>c__DisplayClass30_0.<<InvokeEodCallsForExchangeAsync>b__0>d.MoveNext() in C:\repos\kyna\dotnet\libs\Kyna.Infrastructure\DataImport\EodHdImporter.cs:line 418
--- End of stack trace from previous location ---

kyna-data-import completed in 19.39 minutes

Vic@Archie MINGW64 /c/bin/kyna-imports
$ ./kyna-data-import.exe --info
Source        : eodhd.com
Daily Limit   : 100,000
Used          : 92,286
Available     : 7,714
Requests Date : 2024-02-06

Some things to consider:

https://stackoverflow.com/questions/72275/when-should-the-volatile-keyword-be-used-in-c

https://learn.microsoft.com/en-us/archive/msdn-magazine/2005/october/understanding-low-lock-techniques-in-multithreaded-apps

[vicsharp-shibusa]

I got rid of the volatile properties and the usage of Interlocked and used lock (_locker) instead. On today's test, I got:

Fundamentals for LASR.US
Fundamentals for LATG.US
Insider Transactions for MCAC.US
Fundamentals for LAUR.US
Kyna.Infrastructure.DataImport.ApiLimitReachedException: Available credits (-18) is less than expected cost of 10 (InvokeFundamentalsCallAsync)
   at Kyna.Infrastructure.DataImport.EodHdImporter.CheckApiLimit(Nullable`1 expectedCost, String caller) in C:\repos\kyna\dotnet\libs\Kyna.Infrastructure\DataImport\EodHdImporter.cs:line 998

Vic@Archie MINGW64 /c/bin/kyna-imports
$ ./kyna-importer.exe --info
Source        : eodhd.com
Daily Limit   : 100,000
Used          : 97,528
Available     : 2,472
Requests Date : 2024-02-07

I think my next step will be pause the program and call the User endpoint again - and reset my counts - before throwing the ApiLimitException.

[vicsharp-shibusa]

I believe I have this resolved as of commit 9c6b9f3.

I refactored the CheckApiLimit function to InvokeUserCallAsync before throwing the exception. I also changed the signature on InvokeUserCallAsync and did a bit of refactoring to prevent a doubling of the lock.

    private void CheckApiLimit(CancellationToken cancellationToken, int? expectedCost = null, [CallerMemberName] string caller = "")
    {
        lock (_locker)
        {
            // double check
            if ((expectedCost == null && _available < 1) || (expectedCost != null && _available < expectedCost))
            {
                InvokeUserCallAsync(cancellationToken, true).GetAwaiter().GetResult();
            }
            if (expectedCost == null && _available < 1)
            {
                throw new ApiLimitReachedException(caller);
            }

            if (expectedCost != null && _available < expectedCost)
            {
                throw new ApiLimitReachedException($"Available credits ({_available}) is less than expected cost of {expectedCost} ({caller})");
            }
        }
    }