My results FIT Counts not showing for ccar

[CCarvell]

on the My results page https://ukpoms.org.uk/my-results I have selected 2021, FIT Count app, Apply Filters - and nothing is showing - under the table it states "No information available" - but I have uploaded 5 counts from the app so far.

This may relate to issue #112 I'm not sure?

@andrewvanbreda and @BirenRathod note that @kitenetter is on leave this week, so please let me know if there are any website issues that prove problematic!

[andrewvanbreda]

@BirenRathod @CCarvell I checked another user and they are fine. But I saw the Warehouse database was having problems with NPMS this morning. I also noticed on the Fit Count Map that the attributes are not being filled in this morning, this means also likely to be a Warehouse issue. @BirenRathod Are you the scheduled tasks running ok on the Warehouse at the moment?

[BirenRathod]

@andrewvanbreda Yes, it running but it has got warning about not finishing task in time. May be left with some pending jobs.

[andrewvanbreda]

Hi @CCarvell Are your records appearing on the general Fit Count Map? If so, could you let me know some of the ID numbers associated with those records.

[CCarvell]

@andrewvanbreda here are a few of my IDs - I think!

14499641 14457484 14388215

[andrewvanbreda]

@CCarvell Ok thanks. Will look into this

[andrewvanbreda]

@BirenRathod Hi Biren, Could you run this against live for me please. Cheers.

select created_by_id from indicia.samples where id in (14499641,14457484,14388215);

[BirenRathod]

@andrewvanbreda All returned with 301649.

[andrewvanbreda]

@CCarvell @BirenRathod Hi Claire, The website is behaving correctly given the data it has been given. The problem is with the data in the general Warehouse database (which also holds records from iRecord etc). Essentially the problem is there are two accounts for yourself. One is attached to the app and one is attached to the website. That explains why the website doesn't see your records under My Records.

Now the reason this has happened is that the central database account attached the website is marked as deleted. Do you know how this may have happened? Have you removed yourself from any websites like iRecord recently? I do not know why that would have happened.

I can fix this by switching your PoMS website account to point at the active account attached to the app if you want me to go ahead with that course of action.

[BirenRathod]

@CCarvell Because your account is already exists in the warehouse, if you register via app, the app fails to re-register and website fails to send email. This could be affecting to other users too.

[andrewvanbreda]

@BirenRathod Hi Biren, I am not sure I understand this situation. If the app is working correctly, why is the registration failing? it should just link to the same account already in the Warehouse database.

The problem Claire is having to this is different. I think the order of events for Claire is something like this

1. Registration on website (which is then linked to the Warehouse)
2. For some reason this account is deleted from the Warehouse.
3. Registered with app, but because the previous Warehouse account is in deleted state, a new Warehouse account is created. This means the website and app are then pointing at different accounts.

@CCarvell : I think I just need to know two things

1. Do you have any idea how this account might of been deleted?
2. Let me know if you are ok with me pointing your website account at the active app account

[BirenRathod]

@andrewvanbreda Registration fails when somebody try to register with existing email address. Because on the warehouse, only one account per email address accepted. When somebody try to register via app, it is creating account on Drupal website also, that is fine for Drupal but not good for warehouse if account is already exists.

Also another problem is - if Registered account on app been deleted on warehouse and not deleted on the website that means existing account on website still link with deleted warehouse user id.

Need to find out first, how account has deleted? if deleted , have they deleted on App, website and warehouse?

[andrewvanbreda]

@BirenRathod If this is what app is doing it is incorrect, it should link to existing account. This sounds all wrong to me. Any account deletion on app should not delete the Warehouse one as it is already in use. That should be handled by the warehouse. This does not sound right to me again. Anyway, let's wait to see what Claire says.

[BirenRathod]

@andrewvanbreda I don't know yet what has done on app. so let's get explanation from Claire.

[CCarvell]

@andrewvanbreda @BirenRathod @kazlauskis The below is relevant to both PoMS website and app:

I think this is what I have done, but order of events is tricky as I don't know whether test versions/ beta versions have got in the way:

1. Registered on the website ukpoms.org.uk using ccar@ceh.ac.uk and later given administrator status by @andrewvanbreda on 28/03/2021 - then had to re-set my password yesterday
2. Registered with the app using ccar@ceh.ac.uk but a different password to the password originally provided on the website

This is the advice Martin and I have been giving volunteers, it's on the webpage too: https://ukpoms.org.uk/fit-count-app

"If you have already registered with the FIT Count app it is best to use the same email address for the website, and then you should be able to see all your PoMS surveys in once place. "

so I thought we use the same email address but a different password.

Should it be the same password for both app and PoMS website - ie. to get the same single Warehouse account? If so, we may urgnetly need to update our advice, and will lots of people have two accounts when only one is needed?

I can also see potential problems when we come to release the FIT Count app in other countries - we need a Warehouse just for the app (hence the 'headless' app site and deals with user registrations) and a separate Warehouse for UK PoMS users, or Cyprus, or S American projects etc, I think?

Therefore Andy to answer your questions:

1. Do you have any idea how this account might of been deleted? - NO but I did re-set my password for the PoMS website
2. Let me know if you are ok with me pointing your website account at the active app account - OK yes! but will that affect passwords? and what about other countries using the app that are not part of UKPoMS?

Do we need a discussion about this over teams/ zoom? perhaps best to wait till @kitenetter is back but any fixes to help users of the app now would be great.

Claire

[CCarvell]

ps. @andrewvanbreda @kazlauskis note that I just logged out of my app account and back in again. Now I cannot see any of my Uploaded surveys showing on the app - it says "No uploaded surveys" - will post this on the app GitHub too.

[kazlauskis]

@CCarvell the passwords can be different on the websites, as long as the emails are the same the accounts should be linked in the Warehouse.

Any data, including the uploaded surveys, will be removed from the device after the user logs out or reinstalls the app. When logging out there is an option to leave the local data - uncheck the 'Discard local data' checkbox.

[andrewvanbreda]

@CCarvell In answer to your question 2. This change won't affect anything other than the UKPOMS website. It definetely needs to be done anyway, you can't have the website continue with a deleted Warehouse account.

As far as I am aware the website is operating correctly. Really my work was to setup the website so that shows the app data, and allows it to be edited as required. I believe this situation is correct. I haven't really had anything to do with the app development so can't comment on that. However, the actual deletion of account is something the Warehouse does, so the app may well not be at fault either.

Let me check when that happened and we might get an idea. I will get back to you shortly.

[andrewvanbreda]

@CCarvell @kazlauskis @BirenRathod OK, I have checked this. The account in question was actually deleted in 2014. This is interesting as it may explain another issue we had. This suggests actually it is neither the website or app that is causing the problem, but when the website/app talks to the Warehouse to link to existing accounts (such as iRecord), the Warehouse is including deleted accounts. I will check this john if know problems in the area. Will get back to you.

[andrewvanbreda]

@CCarvell Claire your UK PoMS account is fixed. I can now see 5 My Records on it. Checking with John about possible known issues with Warehouse.

[andrewvanbreda]

@CCarvell @kazlauskis @BirenRathod @DavidRoy John had a look at this and made a good observation that I didn't notice, that actually the creation and deleted (updated_on) date on that account is the same. This means that when it was deleted, it wasn't 2014, but simply that whatever deleted i didn't set the date correctly. Also looks like the code for this issue might have caused #106 It does sound like a Warehouse problem though (so the app is not at fault at all). @DavidRoy, John has said this to me to check with you to "confirm this is a high enough priority for me to spend a few hours of my time on, then I'll be able to take a look"

[CCarvell]

@andrewvanbreda great, thank you for fixing my account - I can now see my 5 app-generated FIT Counts on the website!!

However, when I select Survey: FIT Count, then Apply filters, no data is showing - yet this should pick up the counts I entered into iRecord either this year (which I did in April) or in previous years - right? Is this linked to issue #106 as well?

@DavidRoy may still be on leave but I would support @andrewvanbreda in saying this is high priority please, if we are to avoid putting PoMS or app users off altogether. I will hold off putting out a message to volunteers for now BUT if it's not fixed by Friday and if we get more queries, I think we should agree a short form of words to put on the website and Tweet to explain that we are working to resolve these technical issues?

[DavidRoy]

can you confirm if this is a wider problem or isolated to Claire's account because of issues between the website and app account registration. If it's not a wider problem then this is not necessarily high priority to resolve

[DavidRoy]

There are no blocked user accounts and active registrations in the last few days. FITcounts are being submitted today

[andrewvanbreda]

@CCarvell OK, that problem makes sense actually as those records are going to be pointing at the old ID number. Will get Biren to look in the database.

@DavidRoy Yes it is a suspected problem in the general Warehouse code during registration. It is doing something funny at this point. I do not think it is specific to Claire, although she may have had the unusual circumstances needed for it to happen. It clearly isn't happening all the time, but it is the Warehouse that has control over this situation, and PoMS doesn't use any custom registration code. Obviously can't be 100% certain about this until it is actually investigated. The exact wording John used are as follows so you can decide on that

"If I can reproduce the problem then they will be easier to fix. I've looked at the appropriate code (modules/indicia_svc_security/helpers/user_indentifier.php and I think that the deleted flag on the users table IS checked when it retrieves a user to link to (around line 148 and 160).

But, user 82207 has the updated on set to the same as created on, so I doubt that it was actually deleted in 2014. It's more likely the deletion didn't set the updated on flag. In fact the merge users code at the bottom of this page looks like it might have that effect.

I think this area of code hadn't been touched in a long while and I'd really like to review the test cases around it to make sure it is robust. As that will take a few hours, can you ask the powers that be to confirm this is a high enough priority for me to spend a few hours of my time on, then I'll be able to take a look."

[andrewvanbreda]

@BirenRathod Hi Biren, Whenever you get a sec, could you get me these counts from the database. Cheers

select count(id) from indicia.samples where created_by_id = 82207 and deleted=false; select count(id) from indicia.occurrences where created_by_id = 82207 and deleted=false;

[BirenRathod]

@CCarvell This has got nothing to do with migration. This has generated from third party service used with Pantheon to deliver email. By looking at the error there is one possibility is that user might have deleted this email address.

[BirenRathod]

@BirenRathod Hi Biren, Whenever you get a sec, could you get me these counts from the database. Cheers

select count(id) from indicia.samples where created_by_id = 82207 and deleted=false; select count(id) from indicia.occurrences where created_by_id = 82207 and deleted=false;

@andrewvanbreda First query result is 20 Second query result is 100

[andrewvanbreda]

@BirenRathod Thanks Biren

@CCarvell Looks like you have 20 samples, 100 occurrences which are attached to the user account that had become deleted. These might be associated with other websites such as iRecord. I will have to have a look at this because it might actually be easier to undelete the old account if mistakenly deleted and use that. I don't think that is something I will get around to today but will let you know

[andrewvanbreda]

@kitenetter I just got a mail into my email account saying you commented on this thread, but I cannot see the comment. Leave open until I get chance to fix Claire's account.

It would not be advisable for user's to be able to change the Warehouse email address as that would cause extra confusion, as the Warehouse only holds 1 email address per person, but each of their websites can have their email address altered independently at any time......I never said life was easy :) I will close when ready

[BirenRathod]

@kitenetter I don't have a problem if you close this.

[kitenetter]

Trying to catch up with this thread, and am finding it very confusing - apologies if I now add to the confusion.

There is mention above of user ID 82207, which appears to relate to a deleted warehouse account for Claire. However, I cannot find any PoMS records attached to that account, I can only find BeeWalk records (survey ID 178). Not sure if it is correct that the account for user ID 82207 should be showing as deleted, but it doesn't appear to be related to PoMS.

Claire has another user account, user ID 133147, that was originally linked to iRecord, and that was used for entering FIT Counts on iRecord. That user ID is also showing as deleted in the warehouse, although it still appears to be a live Drupal account on iRecord: https://www.brc.ac.uk/irecord/user/31338

So as far as I can see it is user ID 133147 that needs to be undeleted in the warehouse, and then linked to user ID 301649 (which is Claire's FIT Count app user ID).

I find it worrying that people can have an active iRecord Drupal account, that they can log in to, but that is linked to a deleted warehouse user ID. Surely if an account is deleted in the warehouse it should also be deleted from the Drupal site?

I don't know what if any action is required for user ID 82207, but it doesn't appear to relate to PoMS.

I can't see any implications for PoMS beyond the issues with Claire's accounts.

[andrewvanbreda]

@kitenetter I need to fix Claire's account, but quickly in answer to your question about the Warehouse. John is going to look into this. The system doesn't delete Drupal accounts based on Warehouse deletion. That would be very difficult I think. However the Warehouse account should not have been deleted in this case, it is the Warehouse that incorrectly deleted automatically. This is not a PoMS issue (well not directly). John is going to look into this situation.

[andrewvanbreda]

@kitenetter and in terms of the confusion.....I think the only thing you need to know is John is going to look at Warehouse behaviour and I need to fix claire's account.

[andrewvanbreda]

Hi @BirenRathod ,

Have indeed got very confused with this, as we seem now to be dealing with 3 accounts.

So I am going to start from scratch. Biren could you run these 9 queries when you get chance, as there is now an extra user ID to take into account after Martin's extra information. Cheers

select count(id) from indicia.samples where created_by_id = 82207 and deleted=false; select count(id) from indicia.occurrences where created_by_id = 82207 and deleted=false; select deleted from indicia.users where id = 82207; select count(id) from indicia.samples where created_by_id = 133147 and deleted=false; select count(id) from indicia.occurrences where created_by_id = 133147 and deleted=false; select deleted from indicia.users where id = 133147; select count(id) from indicia.samples where created_by_id = 301649 and deleted=false; select count(id) from indicia.occurrences where created_by_id = 301649 and deleted=false; select deleted from indicia.users where id = 301649;

Note to myself: One we have decided which ID to move the data to, we also need to make sure Claire's website accounts are all also pointing to that same ID

[BirenRathod]

@andrewvanbreda Results are in same order as queries stated above. 20 100 true 54 40 true 7 12 false

[andrewvanbreda]

@BirenRathod Thanks Biren Ok that settles it, I think we will need to repoint everything at 301649 which is Claire's only non-deleted account (which is also her fit count app one). Then make sure all her Drupal accounts are point at that.

Will get back to you next week as that will require updates to live db, not a job for late on Friday afternoon in case of problems.

[andrewvanbreda]

Hi @johnvanbreda

Are you able to to have a look at this issue at all? It is very long trhead so if you are confused about anything, maybe best asked me.

Essentially what has happened is we Claire noticed that the app records she had entered were not appearing on the website.

It turns out what happened was that Claire registered on the website, but the linking Warehouse account magically became deleted. So 82207 was showing in the Indicia Warehouse ID field, even though that account was set as deleted on the Warehouse. So she then registered on the app and that linked to a different Warehouse account (presumably a new one). This means that she would indeed not see the records on the website.

Do not worry about all the talk in this thread about correcting her account. I will need to do that after any fix.

I really just need your thoughts about whether there is a bug in the Warehouse causing this and whether you would be able to spend some time to fix.

See David's note above about whether it was a wider problem, I think the implication is that you could spend some time on it as we suspect it is wider issue than just PoMS, but confirm with him if you are unsure. Comment starts "can you confirm if this is a wider problem or isolated".

The deleted account ID is 82207

The ID that is linked to her app account is 301649 She also has another account ID 133147 would was originally used with iRecord.

I think claire is unsure how 82207 became deleted, but that is the number that was originally attached to her website account (it no longer is, it was switched to 301649 to match her app ID).

I am not sure whether she was asked the merge message for multiple accounts when she registered, but you did think that code might have this affect.

Andy

[johnvanbreda]

The API which registers a new warehouse user can sometimes fail to create a new unique user account because there are more than 1 possible existing match. In this case it will ask the user to help work out which one to use and will then merge the 2 possible accounts. This does result in one of the accounts being left with no records and deleted. I've just added a test to the unit testing to be absolutely sure only the correct user is getting deleted and it is. However, even though the code is functioning as intended, this might leave the iRecord app pointing to a now deleted user.

So - I need to do a bit more thinking to work through all the possible scenarios here and work out exactly how they should be handled.

[kitenetter]

@johnvanbreda we've had another similar report from a user who says:

I’m not sure if it’s connected, but around the time I was trying to register on the new POMs website in early June, it seems that records I submit via iRecord via the iRecord App, iButterflies App or directly via the iRecord website are no longer showing up in ‘Explore My Records’. I logged out of the iButterflies app on my phone and appear to have lost at least 39 site records that had apparently been submitted. The last records I can see when I use ‘Explore my Records’ are from the 6th June 2021.

I know while registering on the POMs site it asked to confirm that I used various indica websites and I said yes, but it kept crashing. I’m hoping my records have gone somewhere recoverable?

I ran this query: SELECT p.id AS p_id, p.surname, p.first_name, p.email_address, w.title AS website, uw.created_on AS uw_created, uw.updated_on AS uw_updated, p.created_on AS p_created, p.updated_on AS p_updated, p.deleted AS p_deleted, u.id as u_id, u.created_on AS u_created, u.updated_on AS u_updated, u.deleted AS u_deleted FROM users u JOIN people p ON u.person_id = p.id JOIN users_websites uw ON u.id = uw.user_id JOIN websites w ON uw.website_id = w.id WHERE p.first_name = 'Scott' AND p.surname = 'Shanks' ORDER BY u.id

[results supplied to John via email]

This seems to be another case of the merge process going wrong. Can we relink the accounts and restore the missing records?

Marking as high priority due to missing records. Let me know if you want this as a new issue, or transferring to the iRecord github.

Do we have any way of checking whether similar problems have arisen for other users?

[kazlauskis]

My primary Warehouse account was flagged deleted a few months back when I have accepted to merge my accounts (this was some other project). I do have a few accounts that I use for testing etc so I wasn't too worried that this has happened, and maybe that was a fluke but it seems like it can mess up actual user accounts too.

[johnvanbreda]

@kitenetter I've fixed the data in this instance and I'm working on some code changes to stop it happening again.

[kitenetter]

@johnvanbreda the user is still not seeing their recent records in their "My records" view - is that just taking a while for the updates to fed through, or is there still something that needs updating?

[andrewvanbreda]

@johnvanbreda @kitenetter If this is like Claire's, the Indicia Warehouse ID on the website will need manually updating to the non-deleted ID. I can't check this right now, but can have a look later

[andrewvanbreda]

@johnvanbreda @kitenetter I just checked, and the website account is linked to Indicia user 68198. That user is not in a deleted state on the Warehouse so is correct. If they are not seeing their records, that suggests those records are attached to a deleted Warehouse User record and will need repointing to be for user 68198

[johnvanbreda]

@kitenetter Scott's most recent record in the database is 20842427 and I can see this on his My Records view. Which missing records are you referring to?

[kitenetter]

@johnvanbreda Scott's records are now showing correctly, thanks - they weren't last night, but presumably I was just being impatient in expecting them to appear more quickly.

[kitenetter]

Going back to the original issue, Claire still has 54 samples in the database that are linked to the deleted user ID 133147. Can we relink these to her non-deleted user ID 301649?

[johnvanbreda]

See https://github.com/BiologicalRecordsCentre/iRecord/issues/1110 for long-term resolution discussion as opposed to short-term support.

[johnvanbreda]

@kitenetter I've updated Claire's samples.

[kitenetter]

Thanks @johnvanbreda - I think we can close this one now.