Repeating folders

[marinaomega]

We need to understand why we have repeating folder (FedEMIS, FedEMIS, etc. MIEMIS, MIEMIS, etc. photos, photos)

[maratomega]

Hi @ghachey To fix this bug we suggesting following:

1. Remove the ability to create folders with the application
2. The survey will try to upload to the cloud until it is possible
3. All folders that are now in the cloud, you will need to delete in manual
4. If the poll is unable to load into the cloud for a long time, there may not be a folder where it should be loaded (the folder does not exist in the cloud or it is currently occupied, files are downloaded from another device), so let’s add to debug: storage: show folder functionality, if they are lost for any reason, which is very unlikely

At the moment, the app has the debug folder removal and poll feature: show storage, but be careful because the app does not notify the user that something is now being deleted. Do we need this functionality in Production operating mode?

Waiting for your approval

[ghachey]

@maratomega Before we proceed with this we will need to discuss it as it will require careful coordination with the backend development. I will reach out to you directly for this.

[ghachey]

Let's schedule a call to discuss this more in details.

[maratomega]

Hi @ghachey did you/managers have a chance to catch this issue again?

[ghachey]

@maratomega this issue persist. We end up with many duplicates still. But the suggested fix seems to drastic at the moment when everything otherwise working. We need to make sure we understand the issue carefully. This is one of the reasons I requested we include a tablet physical ID to the files to observe the source of duplicates. If you can gather more details as to why this is happening we can document here the findings. A reminder to not touch the production accounts when testing, etc.

[maratomega]

Hi @ghachey Still wasn't able to replicate the issue on our side.

Can you please clarify the current logic of the Survey Folders on Google drive? Is it required to create a new folder every year? New folder for every region?

We still offering to try to remove the feature to create folder via app. To have a hard coded folder where all the surveys will upload. If the issue still persists then there gonna be a need to check it in other places, like backend.

[ghachey]

@maratomega We have duplicates of files and of folders. This issue persist and becomes a pain in production with lots of files.

I am not sure where you got the requirement to create a new folder every year? Are you actually doing this? This is absolutely not necessary. Let's get on a call to clarify this. There should be only a single FEDEMIS folder where the surveys go in. And there should be a single FEDEMIS/photos where photos file go in. Currently we end up with structures like this.

-FEDEMIS --photos --photos --photos -FEDEMIS --photos --photos --photos --photos --photos --photos -FEDEMIS --photos --photos etc.

What do you mean by new folder for every region?

Each country has its own cloud service account. For example,

Marshall Islands cloud service account should have MIEMIS folder MIEMIS/photos (i.e. photos folder in MIEMIS)

FSM cloud service account should have FedEMIS folder FedEMIS/photos (i.e. photos folder in FedEMIS)

etc.

Exactly where do you actually create folders? This is what we need to look at. Please tell me which file and source code line you create folders so I can audit.

Let's get on a call to make all this clear.

[ghachey]

It is possible that removing the ability to create new folders and hard coding the simple following folder structure may help in reducing and possibly eliminating this duplicate issue that persist.

The standard structure in ALL google service accounts (regardless of mode such as production, devleopment or training).

CONTEXT CONTEXT\photos

where context is the deployed country. Examples.

FEDEMIS FEDEMIS\photos

or

MIEMIS MIEMIS\photos

or

KIEMIS KIEMIS\photos

etc.

A reminder to not touch anything in Production here. Like do not attempt to cleanup existing duplicate folders, etc. Simpler work out from the new Training mode and/or the Development (test) accounts.

[ghachey]

@maratomega

Hi @ghachey To fix this bug we suggesting following:

1. Remove the ability to create folders with the application

Yes.

2. The survey will try to upload to the cloud until it is possible

Yes.

3. All folders that are now in the cloud, you will need to delete in manual

Yes. The cleanup will be manual in the future. We will cleanup first the Development, then eventually also the Production.

4. If the poll is unable to load into the cloud for a long time, there may not be a folder where it should be loaded (the folder does not exist in the cloud or it is currently occupied, files are downloaded from another device), so let’s add to debug: storage: show folder functionality, if they are lost for any reason, which is very unlikely

I am unsure what you mean here? Can it simply not just fail to sync and the syncing status icon will just reflect this with red cloud as we normally have now? And then we can "force" a new sync in a short while to see if the sync works a little later (folder no longer occupied)?

At the moment, the app has the debug folder removal and poll feature: show storage, but be careful because the app does not notify the user that something is now being deleted. Do we need this functionality in Production operating mode?

I really do not understand what you mean by this above paragraph. If some change is proposed can you please provide more details in another comment?

[ghachey]

This was such a subtle bug that I will leave it open for now and observe for months. There is some cleanup to do in the service accounts to remove duplicate folders but we can only do this when all data has been validated and fully loaded into the system. So it can't be fully tested just now.