[issue] Persistent Upload Failure for Photos Taken Without Network Connection

[sync-by-unito[bot]]

Based of this slack message

Capture App keeps showing "Registering N Captures onto blockchain."

┆Issue is synchronized with this Asana task by Unito ┆Created By: Sam

[sync-by-unito[bot]]

➤ Bofu Chen commented:

Increasing the priority to critical because registration is the core and fundamental function in Numbers system.

[sync-by-unito[bot]]

➤ Bofu Chen commented:

Celery and Redis task queues are empty, which means that there is no pending task in the backend.

[sync-by-unito[bot]]

➤ Sam commented:

Bofu Chen, thank you extra info (helps to narrow down debug surface). It means those assets never left the app and most likely issue is on Capture App side. I will check code related to uploader.

[sync-by-unito[bot]]

➤ Bofu Chen commented:

I created 2 testing assets, each with 2 registration logs. The 2nd registration came after the 1st one around 4 - 5 seconds.

https://gist.github.com/bafu/0ebfc6fa56a5aa967427eca978526bd1#file-register-two-testing-assets-txt ( https://gist.github.com/bafu/0ebfc6fa56a5aa967427eca978526bd1#file-register-two-testing-assets-txt )

[sync-by-unito[bot]]

➤ Bofu Chen commented:

Capture App keeps trying to register the last successfully registered asset.

The asset was created on 0321, but the logs before 0325 have been flushed. So, the oldest log of the asset we have is on 0326.

https://gist.github.com/bafu/0ebfc6fa56a5aa967427eca978526bd1#file-tammy-registration-logs-txt ( https://gist.github.com/bafu/0ebfc6fa56a5aa967427eca978526bd1#file-tammy-registration-logs-txt )

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Here is my suggestion (cc Kenny Hung to follow up together).

The current app has a big issue that NO capture can be opened until the uploads are successful. This bring huge risks of data loss if the uploading blocked.

Here is my suggestion to fix the issue:

1. When upload has not yet finished, should allow user to open the photo and click upload
2. When users are online, should allow user to open the photo and show something like "Connect to internet to show more details from blockchain"

If this is fixed, then we should be able to avoid the issue. I can help test with Google Play Beta.

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Kenny Hung I think this is critical than most of the issues because if users cannot access their Captures before uploading finishes, that's a critical data loss issue. If you agree, please help arrange the task asap. Thanks!

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Tammy Yang (cc Bofu ChenSam) Okay, add this into 0410 sprint.

BTW, Could you help to add reproduce steps?

I have tried to reproduce it, but I can't.

Here is my steps

1. Close the connection
2. Take a photo on Capture App
3. Open the connection
4. Waiting for uploading finish. (On discussion it should can't be finished.)

About the suggestion, on ionic-v230321 already have a similar task ( https://app.asana.com/0/1201016280880500/1203921730220982/f ), maybe also could fix this task.

[sync-by-unito[bot]]

➤ Tammy Yang commented:

No, no reproduce step, so I have said I will help tested it.

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Scott Yan (cc Tammy Yang ) Currently, there seems to be a user who is experiencing this issue as well. Could you help to check from the backend?

email: akashad8871@gmail.com ( mailto:akashad8871@gmail.com )

[sync-by-unito[bot]]

➤ Scott Yan commented:

on backend check

1. An asset was created on 4/17 and no new assets have been added since then.
2. There are no pending or failed network actions, and all actions show a successful status.
3. The user's asset wallet has enough tokens.
4. no pending asset transation

At the moment, the situation appears to be normal. However, if more detailed information is needed, developer assistance may be required.

[sync-by-unito[bot]]

➤ Kenny Hung commented:

SamScott YanTammy Yang

The further information

Doesn't operate any VPN or background app

Model - Realme XT (RMX1921)

OS - Android 11

It is always connected to Internet

[sync-by-unito[bot]]

➤ Tammy Yang commented:

May need Olga or James Chien to check backend log.

[sync-by-unito[bot]]

➤ Olga commented:

checked assets created by akashad8871@gmail.com ( mailto:akashad8871@gmail.com ), the first 3 assets are created on 4/17 and is uploaded by camera.

[sync-by-unito[bot]]

➤ Olga commented:

These 3 latest assets are

bafybeiad7wckxh3vk7ffoxwmyoha57v7z6cr62h3io3wk2wsd57bjhcoaa

bafybeihzaujztciphq7s64zybs4un5wrq36iwpfd76pkatx7kmlt3mi2ni

bafybeictyyvbxqw4nrzgyct2z6twcfayjjsapk27dll6upecexaz3vdqfq

And their post_creation_workflow_id indicate status is success.

Sam what condition will app show the message "Registering XX Captures onto blockchain."?

[sync-by-unito[bot]]

➤ Sam commented:

Olga

When network is connected and upload is not paused

[sync-by-unito[bot]]

➤ Olga commented:

Sam what's the definition that upload is not paused? I have to know the definition so I can provide more backend support.

[sync-by-unito[bot]]

➤ Sam commented:

Olga, sure.

In capture app user can pause upload process.

When user takes photo it will automatically start uploading it to the backend.

Also user can manually press pause upload and it will stop uploading.

When user pause upload and take photo it will not upload to the backend and will sit in capture app. When user click unpause again it will loop through upload queue and retry to unpload unfinished uploads.

[sync-by-unito[bot]]

➤ Olga commented:

Sam thanks for the information. Do the unfinished uploads mean the assets aren't registered in backend, the api https://dia-backend-dev.numbersprotocol.io/api/v3/redoc/#operation/assets_create ( https://dia-backend-dev.numbersprotocol.io/api/v3/redoc/#operation/assets_create ) doesn't return 200?

The user's latest assets are created on 4/17 Comment by @Olga on [issue] Persistent Upload Failure for Photos Taken Without Network Connection ( https://app.asana.com/0/0/1204297132869604/1204468031384777/f ) and the user doesn't call the api since 25/Apr/2023:08:01:06 +0000.

[sync-by-unito[bot]]

➤ Sam commented:

Olga in capture app if endpoint does not return 200 it will retry ( https://github.com/numbersprotocol/capture-lite/blob/9f490be95cf0f1c436a284ca13cacef77ee0a0b7/src/app/shared/dia-backend/asset/uploading/dia-backend-asset-uploading.service.ts#L150 ) again.

But I think

and the user doesn't call the api since 25/Apr/2023:08:01:06 +0000.

is valuable information maybe I need to check upload service in capture app again.

[sync-by-unito[bot]]

➤ Sam commented:

Kenny Hung, do you have a device with upload fail state?

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Sam Currently no, Tammy Yang has previously encountered this issue.

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Tammy Yang (cc Sam ) Can we change this task priority to high & push it to next sprint?

Because

1. When v230413-ionic public, though user may still can't complete register asset, but user can enter asset page & save them on local. They won't lose these captures.
2. QA still can't reproduce this task.
3. v230516-ionic is blocked by this task.

If we confirm, Sam could release v230516-ionic today.

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Ok

[sync-by-unito[bot]]

➤ Sam commented:

Kenny Hung, can we move this task to sprint after patch sprint?

Reason is in v230613-capture-app-ionic-launch ( https://app.asana.com/0/0/1204495833338689 ) this task [Sprint 2] Upload video(SHORT) ( https://app.asana.com/0/0/1204202515424193 ) took most of the time due to unexpected complexities Comment by @Sam on [Sprint 2] Upload video(SHORT) ( https://app.asana.com/0/0/1204202515424193/1204835672867627/f ).

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Tammy YangSam

I push this task to next sprint. Because of main feature of v230613-capture-app-ionic-launch ( https://app.asana.com/0/0/1204495833338689 ) is [Sprint 2] Upload video(SHORT) ( https://app.asana.com/0/0/1204202515424193 ). We should release it in the current release.

[sync-by-unito[bot]]

➤ Sam commented:

Kenny Hung, (cc: Tammy Yang):

Need confirmation.

According to Comment by @Kenny Hung on [issue] Persistent Upload Failure for Photos Taken Without Network Connection ( https://app.asana.com/0/0/1204297132869604/1204600124434831/f ) I just need to add download button in details page?

[sync-by-unito[bot]]

➤ Kenny Hung commented:

SamTammy Yang

No, for UI part, we don't need to adjust it. This task should be "make sure the asset could be registered completely", even user cut the connection during the processing.

[sync-by-unito[bot]]

➤ Sam commented:

Kenny Hung, yes thats the issue because it's not reproducible it's hard to fix. So first we need to find a way to reproduce and it will be easier to fix.

However potential issue is capture app does not try to re-register asset after failure. Comment by @Olga on [issue] Persistent Upload Failure for Photos Taken Without Network Connection ( https://app.asana.com/0/0/1204297132869604/1204475992010108/f )

The user's latest assets are created on 4/17 Comment by @Olga on [issue] Persistent Upload Failure for Photos Taken Without Network Connection ( https://app.asana.com/0/1201016280880500/1204297132869604/1204468031384777 ) and the user doesn't call the api since 25/Apr/2023:08:01:06 +0000.

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Sam do we know the reason why? There is a pause button and the upload should restart when users press it on and off. James Chien do you recall?

[sync-by-unito[bot]]

➤ James Chien commented:

I don’t think we’ve ever found the root cause of the issue

[sync-by-unito[bot]]

➤ Sam commented:

Tammy Yang,

• I don't know the reason why yet.
• Yes it should restart uploads and the logic ( https://github.com/numbersprotocol/capture-lite/blob/02ae4e74227cfc46f537ad1ead006ddf27d5eb18/src/app/shared/dia-backend/asset/uploading/dia-backend-asset-uploading.service.ts#L67 ) didn't changed since long time. So it suppose to work as it worked before.

Since this task is not easily reproducible I have this proposal?

• Create internal release that will have a button re-upload in capture details page.
• Do firebase release (not visible to Public) so Tammy can download it
• When Tammy downloads It should update app without losing any data (like login and not uploaded captures)
• Then Tammy choose any capture that stuck uploading
  • go to capture detasils page
  • click re-upload ( it will do something like print capture state or try to re-add to upload queue not sure yet)
  • and from there see what we can do

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Agree with the proposal, let's do it (cc Kenny Hung ) However, I have already re-install the App (yes, then all data was lost), so I won't be able to help with the test.

Does QA have a way to reproduce it and test the fix?

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Tammy YangSam Wait, I review all the comments on this task. I found the initial Tammy's comment ( https://app.asana.com/0/0/1204297132869604/1204307951566766/f ) said,

Here is my suggestion to fix the issue:

1. When upload has not yet finished, should allow user to open the photo and click upload
2. When users are online, should allow user to open the photo and show something like "Connect to internet to show more details from blockchain"

Maybe the current status is enough for user? (Because #1 & #2 are done.) I worried about if we add re-upload button, it may let the user confuse. (Because mycapture tab shows they're uploading.)

Answer about the reproduce & test.

Current QA still can't reproduce, I also try update app during registering many asset, and the data is still here.

I could ask Aaren how about the current status of that user.

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Kenny Hung where is the "button to upload"? I think your #1 matches Sam's suggestion "Create internal release that will have a button re-upload in capture details page." and it has not yet implemented.

[sync-by-unito[bot]]

➤ Sam commented:

Tammy Yang, Kenny Hung.

Nothing is implemented, there were no coding changes related to this task. And before adding upload button or any fix I think we still need to somehow reproduce and see the root cause.

Adding button only make sense if we have a device with stuck uploads (aka reproducible state).

[sync-by-unito[bot]]

➤ Tammy Yang commented:

Kenny Hung if no one can reproduce the issue, I suggest we push the task to Medium priority and keep monitoring it until we can reproduce it.

[sync-by-unito[bot]]

➤ Kenny Hung commented:

Tammy Yang (cc Sam)

oh, I think I just misunderstand Sam's comment.😅

Yes, I agree. I also check again the latest version. Currently, users could enter to asset page even the asset isn't registered completely or offline, but we don't have "click upload" yet.

So, I list the tasks below

1. Add [FR] add status bar in ionic page which is registering. ( https://app.asana.com/0/1201083422707776/1204922225660934/f ) (Not sure what is the UI/UX for "click upload".)
2. [Issue] Offline Image/Video Content Access Issue After Logout and Login ( https://app.asana.com/0/1201016280880500/1204547665570170/f )

Suggest when we finish these two tasks in the future sprint and I'll move this task to qa verify & decrease the priority.

The perfect status should be like Sam said we found a way/device with stuck uploads (aka reproducible state).