ema downloader exit code is 0Xfffff

sergiocontrino commented 3 months ago

it looks like it is not ending well, to check!

michelescarlato commented 3 months ago

to be investigated F:\MDR_Logs\euctr\DL euctr 2024-08-19 162838 USING EMADL.xml.log

19/08/2024 : 16:28 :   ****** DOWNLOAD ******

19/08/2024 : 16:28 :   **** Set up ****
19/08/2024 : 16:28 :   Download event Id is 102142
19/08/2024 : 16:28 :   
19/08/2024 : 16:28 :   Source_id is 100159
19/08/2024 : 16:28 :   Type_id is 103
19/08/2024 : 16:28 :   File name is F:\MDR_Sources\EMA\DL\EMADL.xml

19/08/2024 : 16:28 :   **** UNHANDLED EXCEPTION ****
19/08/2024 : 16:28 :   
19/08/2024 : 16:28 :   +++++++++++++++++++++++++++++++++++++++
19/08/2024 : 16:28 :   19/08/2024 : 16:28 :   ***ERROR*** MDR_Downloader application aborted

19/08/2024 : 16:28 :   Index and length must refer to a location within the string. (Parameter 'length')

19/08/2024 : 16:28 :      at System.String.ThrowSubstringArgumentOutOfRange(Int32 startIndex, Int32 length)
   at System.String.Substring(Int32 startIndex, Int32 length)
   at MDR_Downloader.euctr.EMA_Controller.ObtainDataFromSourceAsync(Options opts, Source source) in C:\Users\ecrinadmin\source\repos\MDR_Downloader\SourceSpecific\euctr\EMAController.cs:line 50
   at MDR_Downloader.SourceController.ObtainDataFromSourceAsync(Options opts, Source source, DLEvent dl) in C:\Users\ecrinadmin\source\repos\MDR_Downloader\TopLevelClasses\SourceController.cs:line 14
   at MDR_Downloader.Downloader.RunDownloaderAsync(Options opts, Source source) in C:\Users\ecrinadmin\source\repos\MDR_Downloader\TopLevelClasses\Downloader.cs:line 89
   at Program.<Main>$(String[] args) in C:\Users\ecrinadmin\source\repos\MDR_Downloader\TopLevelClasses\Program.cs:line 67
19/08/2024 : 16:28 :   +++++++++++++++++++++++++++++++++++++++
19/08/2024 : 16:28 :   

19/08/2024 : 16:28 :   **** Closing Log ****

michelescarlato commented 3 months ago

@sergiocontrino Sergio, I added a try-catch to allow the execution to terminate with an exit 0 (as the Windows Task scheduler requires to avoid the exit code 0Xfffff).

It seems to be working.

On the other hand, I was trying to improve the error message with some unique ID related to the parsed entry, but I couldn't find it (at the moment). The function takes in input the class objects source and opts, and this is their content when the error arises:

id: 100159    // source ID - euctr
source_type: study
preference_rating: 39
database_name: euctr
repo_name: EMA
db_conn: 
uses_who_harvest: False
harvest_chunk: 100
local_folder: F:\MDR_Data\euctr\
local_files_grouped: False
grouping_range_by_id: 0
local_file_prefix: EU
 and options: 
SourceId: 100159
FetchTypeId: 103
dl_id: 102148   // download event id
CutoffDateAsString: 
CutoffDate: 
EndDateAsString: 
EndDate: 
FileName: F:\MDR_Sources\EMA\DL\EMADL.xml
OffsetIds: 
AmountIds: 
SkipRecentDays: 
NoLogging: 
StartPage: 
EndPage: 
FocusedSearchId: 
PreviousSearches: System.Int32[]
previous_dl_ids: 
PubMedPubs: 
.

So at the moment, I am leaving the error message in the log in this way:

 $"Incorrect date format in one of the entries. Default date will be used. Detail: {ex.Message}");

Which produces:

20/08/2024 : 17:29 :   ****** DOWNLOAD ******

20/08/2024 : 17:29 :   **** Set up ****
20/08/2024 : 17:29 :   Download event Id is 102151
20/08/2024 : 17:29 :   
20/08/2024 : 17:29 :   Source_id is 100159
20/08/2024 : 17:29 :   Type_id is 103
20/08/2024 : 17:29 :   File name is F:\MDR_Sources\EMA\DL\EMADL.xml
20/08/2024 : 17:29 :   
20/08/2024 : 17:29 :   +++++++++++++++++++++++++++++++++++++++
20/08/2024 : 17:29 :   20/08/2024 : 17:29 :   ***ERROR*** Incorrect date format in one of the entries. Default date will be used. Detail: Index and length must refer to a location within the string. (Parameter 'length')
20/08/2024 : 17:29 :   +++++++++++++++++++++++++++++++++++++++
20/08/2024 : 17:29 :   
20/08/2024 : 17:29 :   20 studies checked, 20 downloaded
20/08/2024 : 17:29 :   40 studies checked, 40 downloaded
20/08/2024 : 17:30 :   60 studies checked, 60 downloaded
20/08/2024 : 17:30 :   80 studies checked, 80 downloaded
20/08/2024 : 17:30 :   100 studies checked, 100 downloaded
20/08/2024 : 17:30 :   120 studies checked, 120 downloaded
20/08/2024 : 17:30 :   140 studies checked, 140 downloaded
20/08/2024 : 17:31 :   160 studies checked, 160 downloaded
20/08/2024 : 17:31 :   180 studies checked, 180 downloaded
20/08/2024 : 17:31 :   200 studies checked, 200 downloaded

20/08/2024 : 17:31 :   **** Download Result ****
20/08/2024 : 17:31 :   Source Id: 100159
20/08/2024 : 17:31 :   Download Event Id: 102151
20/08/2024 : 17:31 :   Start time: 20/08/2024 17:29:14
20/08/2024 : 17:31 :   End time: 20/08/2024 17:31:35
20/08/2024 : 17:31 :   Records checked: 204
20/08/2024 : 17:31 :   Records downloaded: 204
20/08/2024 : 17:31 :   Records added: 0

20/08/2024 : 17:31 :   **** Closing Log ****

michelescarlato commented 3 months ago

https://github.com/ecrin-github/MDR_Downloader/pull/1 This PR fixes the issue.

michelescarlato commented 3 months ago

Double-check that the event comes back with 0x0 Last run result and it did.

michelescarlato commented 3 months ago

@sergiocontrino Sergio, I just noticed that the original repository has an important commit that ecrin-github/MDR_Downloader never imported.

It is concretely affecting the EMA repository. You can have a look here:

https://github.com/ecrin-github/MDR_Downloader/compare/master...scanhamman%3AMDR_Downloader%3Amaster

As a very important improvement, I also noticed that .NET has been updated to version 8 (since version 7 is no longer supported).

If you consider importing this commit, it would require time to evaluate and test the changes.

PS is the Test environment still up? I cannot reach it.

sergiocontrino commented 3 months ago

@michelescarlato thank you for the great work. the modifications in the original code seems to be related to the ctis source (tbc) we need to add. let's check with @iPr0ger tomorrow

michelescarlato commented 3 months ago

Great Sergio.

I also noticed that MDR Harvester has the same situation. Both commits have been done by Steve around 12th April.

On Wed 21 Aug 2024, 08:32 sergio contrino, @.***> wrote:

@michelescarlato https://github.com/michelescarlato thank you for the great work. the modifications in the original code seems to be related to the ctis source (tbc) we need to add. let's check with @iPr0ger https://github.com/iPr0ger tomorrow

— Reply to this email directly, view it on GitHub https://github.com/ecrin-github/MDR_FuiPortal/issues/47#issuecomment-2301239875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACTHW7TUPGKRDXUXWSNOOD3ZSQYBPAVCNFSM6AAAAABMZMPPXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBRGIZTSOBXGU . You are receiving this because you were mentioned.Message ID: @.***>

ecrin-github / MDR_FuiPortal

ema downloader exit code is 0Xfffff #47