Import template update

[Chinedar]

Urgent need for the improved import functionality on the platform. Data dictionary update and a sample of an already populated import template would be very helpful.

[Iheanacho2027]

Please this is very important

[MateStrysewske]

@Chinedar Please specify what you mean with "improved import functionality". Our current priorities for this and the next sprint are to make SORMAS completely corresponding to the Global Goods Maturity Model and to add Congenital Rubella as a new disease. We'll talk about this topic when we set the next priorities afterwards.

[Iheanacho2027]

Following up on this @MateStrysewskeSym Some of the problems associated with the import functionality include i. The “location” sheet of the data dictionary doesn’t have a “value” column which shows the unique style or example for entering variables associated with location (i.e the “person” sheet and all other sheets have the “value” column that gives unique style or example for entering variables while the “location” sheet doesn’t). This is also the same for some other sheets on the data dictionary.

ii. Some fields on the respective sheets of the data dictionary do not have the “value” column which gives unique style or example for entering variables. For example, in the “person” sheet, fields like Sex, ApproximateAgeType, PresentCondition, CauseOfDeath, Disease, DeathPlaceType, BurialConductor, EducationType and OccupationType have the ‘’value” column while the remaining fields do not have. All fields in all the sheets of the data dictionary, should therefore have “value” column which gives a unique style or example for entering variables.

iii. There is no specific guide on how to enter dates in different sheets requiring dates, month of the year in particular i.e should the month of the year be in numbers e.g 01 or letters e.g January ?. A sample value which acts as a guide to enter date variables on every sheet should be given ( for example should it be in dd/mm/yy or otherwise).

iv. The timeframe for checking the quality of variables during import is too much. Sometimes it could take up to 30minutes and end up producing no result. The import functionality should be a smart functionality which should take more than ten minutes to import or identify errors.

Alternatively, there could be a sample import CSV sheet where all variables to be imported are completely-filled with sample values to guide users on the use of the template.

[Iheanacho2027]

Hi Mate @MateStrysewskeSym, whats the update on this?

[MateStrysewske]

@Iheanacho2027 We will review your suggestions in our next Sprint Review (which will be this Friday) and probably add the issue to our next sprint.

[Iheanacho2027]

Hi Mate, due to upsurge of cases coming from Nigerian states where the SORMAS hasnt been deployed and the demand coming from the national institute for all cases to be entered into the system, we request that this be made an urgent priority

[MateStrysewske]

@MartinWahnschaffeSymeda Please talk about this in the next sprint review and include it in the next sprint.

[Chinedar]

@danieltomaba please bring up the suggestions I made to you in the discussion at your next sprint review.

[Iheanacho2027]

@MateStrysewskeSym @MartinWahnschaffeSymeda Pls can you give me an appropriate syntax for inputing "Other health facility" in the health facility column of the import template. I am currently trying to work with the template currently available, i think i may make some progress if i can get an answer to the question i just asked

[MateStrysewske]

@Iheanacho2027 Do you want to import a case that is situated in a health facility that is not part of the system? In that case, you'll have to put either OTHER_FACILITY for a health facility not in the system or NO_FACILITY if the case is situated at home or somewhere else into the healthFacility column and then specify the name/description in the healthFacilityDetails column.

Is that way you meant or did you have something else in mind?

[Iheanacho2027]

Thanks "OTHER_FACILITY" is what i have been looking for

On Wed, Aug 7, 2019 at 3:29 PM MateStrysewskeSym notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 Do you want to import a case that is situated in a health facility that is not part of the system? In that case, you'll have to put either OTHER_FACILITY for a health facility not in the system or NO_FACILITY if the case is situated at home or somewhere else into the healthFacility column and then specify the name/description in the healthFacilityDetails column.

Is that way you meant or did you have something else in mind?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGJ47V5WA2K6W4VGU5DQDLL3XA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YS5AI#issuecomment-519122561, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGIPT7J46PXDEDI754TQDLL3XANCNFSM4HOCY6BQ .

[Iheanacho2027]

Just tried "OTHER_FACILITY" and the system rejected it saying OTHER_FACILITY isnt in the database .

On Wed, Aug 7, 2019 at 3:32 PM Henry Henry iheanachodenwigwe@gmail.com wrote:

Thanks "OTHER_FACILITY" is what i have been looking for

On Wed, Aug 7, 2019 at 3:29 PM MateStrysewskeSym notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 Do you want to import a case that is situated in a health facility that is not part of the system? In that case, you'll have to put either OTHER_FACILITY for a health facility not in the system or NO_FACILITY if the case is situated at home or somewhere else into the healthFacility column and then specify the name/description in the healthFacilityDetails column.

Is that way you meant or did you have something else in mind?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGJ47V5WA2K6W4VGU5DQDLL3XA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YS5AI#issuecomment-519122561, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGIPT7J46PXDEDI754TQDLL3XANCNFSM4HOCY6BQ .

[Iheanacho2027]

I have also tried OTHER_HEALTH_FACILITY previously and it was also rejected

On Wed, Aug 7, 2019 at 3:35 PM Henry Henry iheanachodenwigwe@gmail.com wrote:

Just tried "OTHER_FACILITY" and the system rejected it saying OTHER_FACILITY isnt in the database .

On Wed, Aug 7, 2019 at 3:32 PM Henry Henry iheanachodenwigwe@gmail.com wrote:

Thanks "OTHER_FACILITY" is what i have been looking for

On Wed, Aug 7, 2019 at 3:29 PM MateStrysewskeSym < notifications@github.com> wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 Do you want to import a case that is situated in a health facility that is not part of the system? In that case, you'll have to put either OTHER_FACILITY for a health facility not in the system or NO_FACILITY if the case is situated at home or somewhere else into the healthFacility column and then specify the name/description in the healthFacilityDetails column.

Is that way you meant or did you have something else in mind?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGJ47V5WA2K6W4VGU5DQDLL3XA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YS5AI#issuecomment-519122561, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGIPT7J46PXDEDI754TQDLL3XANCNFSM4HOCY6BQ .

[MateStrysewske]

@Iheanacho2027 Seems like this is a bug. I'll fix it and it will hopefully be resolved in our upcoming release. (see #1259)

[Iheanacho2027]

Alright

On Wed, 7 Aug 2019, 16:04 MateStrysewskeSym, notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 Seems like this is a bug. I'll fix it and it will hopefully be resolved in our upcoming release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGPAYH6QOILOPZTD2E3QDLQA3A5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YWV6Y#issuecomment-519138043, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGJP4XWKLPQI2N6J74LQDLQA3ANCNFSM4HOCY6BQ .

[bernardsilenou]

Hi All, Has this issue been solved? if not then we can tackle it while implementing the basic import function for line-list

[MateStrysewske]

@bernardsilenou No, this is still relevant. I agree that we should do this in one go though.

[Iheanacho2027]

The import function is working fine but just a little issue is needed to be resolved for it to be made perfect. The issue is, when a duplicate is spotted, the session keeps loading for a very long time and never stops. This is shown in the image below

[Iheanacho2027]

Also the import template has no sample information data

[MateStrysewske]

@Iheanacho2027 Samples will be included in the next release that we're working on right now. Concerning the infinite loading issue; does this happen everytime or only on some occasions?

[Iheanacho2027]

The infinite loading happens every time a duplicate is spotted as shown in the image

On Thu, 29 Aug 2019, 08:41 Maté Strysewske, notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 Samples will be included in the next release that we're working on right now. Concerning the infinite loading issue; does this happen everytime or only on some occasions?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGJAJDVOEB6BOYR4VP3QG54TVA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5NSI4A#issuecomment-526066800, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGJRRDE67W2QQDXAK2LQG54TVANCNFSM4HOCY6BQ .

[Iheanacho2027]

An update on the import functionality from the latest release

1) There is no column variable for "Pathogen test requested" on the import template 2) There is a bug whereby two LGAs ( Bayo LGA and Gubio LGA in Borno state) are not recognized by the database, i have confirmed that they are properly matched on the district column with the proper region of the import template, I have also confirmed from the database that i properly included the right syntax on the import template. 3) The infinite loading i complained of, in my last comment still exists. I cant comprehensively say if this is due to internet connection but i believe its due to database capacity issues

[MateStrysewske]

@Iheanacho2027

1) Do you actually need this? It's more of a variable that mirrors the internal process of SORMAS; if you're importing samples AND pathogen tests at the same time, is it really relevant whether it's filled in or not? 2) Can you please paste me the exact error message from the error report file? 3) Both doesn't really make sense in my eyes; seems more like an issue with UI threading - I'll investigate this again and see whether I can reproduce it on my local system

[Iheanacho2027]

I agree with you as regards number one, it's not really needed for import.

For number 2, the error given is " district name is not found in the provided region or available on the database", I carried out all necessary checks before logging this

For number 3, I really can't say but it takes a long time to load which could be frustrating. I don't this is due to internet connection because the duplicate checker that occurs when manually creating a case doesn't act the same way.

On Thu, 12 Sep 2019, 06:42 Maté Strysewske, notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027

1. Do you actually need this? It's more of a variable that mirrors the internal process of SORMAS; if you're importing samples AND pathogen tests at the same time, is it really relevant whether it's filled in or not?
2. Can you please paste me the exact error message from the error report file?
3. Both doesn't really make sense in my eyes; seems more like an issue with UI threading - I'll investigate this again and see whether I can reproduce it on my local system

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGKPTCX2NKDXOTLJEZTQJHJFLA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6QXDBA#issuecomment-530674052, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGMLC7KNMR7VTYXSBU3QJHJFLANCNFSM4HOCY6BQ .

[MateStrysewske]

@Iheanacho2027 What exactly did you put in the import file? "Bayo" and "Gubio"? Please verify so I can have a look into it, too.

Concerning the infinite loading; The UI logic that is processed during the import is way more complex and we have to use different UI threads to handle everything. This worked fine so far, but maybe some change broke it. I've already opened an issue for it: #1300

[Iheanacho2027]

[MateStrysewske]

@Iheanacho2027 I BELIEVE there might be a whitespace after the district names that is causing this issue. This is definitely a bug because leading and trailing whitespaces should be removed from the name during the import; Can you please check whether my assumption is correct and try again to import without any whitespaces and see if it works? If so, please report here so I can create a bug issue for this problem.

[Iheanacho2027]

Alright I will

On Thu, 12 Sep 2019, 09:27 Maté Strysewske, notifications@github.com wrote:

@Iheanacho2027 https://github.com/Iheanacho2027 I BELIEVE there might be a whitespace after the district names that is causing this issue. This is definitely a bug because leading and trailing whitespaces should be removed from the name during the import; Can you please check whether my assumption is correct and try again to import without any whitespaces and see if it works? If so, please report here so I can create a bug issue for this problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hzi-braunschweig/SORMAS-Project/issues/1148?email_source=notifications&email_token=AMDDTGKX55OPDRKV57XPMVDQJH4PBA5CNFSM4HOCY6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RC3IQ#issuecomment-530722210, or mute the thread https://github.com/notifications/unsubscribe-auth/AMDDTGIZBXVDQPQ2QFVHA2TQJH4PBANCNFSM4HOCY6BQ .

[Iheanacho2027]

Its now recognizing the two LGAs, with no changes made from me. Probably you fixed something

[MateStrysewske]

@Iheanacho2027 I didn't do anything, but I'm glad it's working now, whatever the reason was. Make sure to get back to us should it happen again.

[Iheanacho2027]

This error exists during import in 1.29.0, pls kindly resolve @MateStrysewskeSym

[MartinWahnschaffe]

As far as I can tell this can be closed. Contacts import is handled here #1154, samples can be imported with the case and the infinite loading is handled in #1300.