search

commons-app / apps-android-commons

The Wikimedia Commons Android app allows users to upload pictures from their Android phone/tablet to Wikimedia Commons
https://commons-app.github.io/
Apache License 2.0
1.03k stars 1.23k forks source link

Add limited connection mode #746

Closed neslihanturan closed 4 years ago

neslihanturan commented 7 years ago

I think most of our users might be travelling abroad and they possibly have limited internet connection. They should be able to select "limited connection mode" to prevent contribution and nearby lists from load (we can display an information string instead). So user can only upload image by using minimal amount of resources. Do you also think such feature would be nice?

misaochan commented 7 years ago

I think Nearby takes a very small amount of bandwidth currently. But you have a point re: contributions (although AFAIK uploading in and of itself takes a whole lot more bandwidth due to the image being full-size, compared to thumbnails for contributions). But, I'm good with this if there is a need for it, at any rate it couldn't hurt. :)

tobias47n9e commented 7 years ago

I think that our current traffic map also needs to get traffic information in addition to the map tiles. Not sure what impact that has.

I think Commons uploads and downloads fall under Wikipedia Zero in participating countries.

Is there a low bandwidth map we can load from mapbox?

nicolas-raoul commented 7 years ago

Image upload is clearly the biggest data plan eater. Far second is probably the gallery, and third maps. The nearby query and categories search take almost nothing.

So I think a "low-bandwidth mode" should:

neslihanturan commented 7 years ago

Upload picture later option is a great idea. We can even create a queue of pictures.

Mapbox supports offline maps if user has selected the area. Plus, I think we need to change our traffic map to pedestrian map in any case.

maskaravivek commented 7 years ago

Yes, it would be great to implement the offline support(upload later). I would be able to help with the flow and sync logic having done it for offline payments. :)

misaochan commented 7 years ago

Sounds good to me. :) Do you guys think it is sufficiently important/easy that we should add it to the renewal proposal? Or if it's not easy, do you think it's sufficiently important that we should remove something else to make way for this?

neslihanturan commented 7 years ago

Hard question. I don't think it is easy, however in my point of view it is quite important.

maskaravivek commented 7 years ago

I agree with @neslihanturan. Its important as in my opinion the feature to queue up pictures for upload and not waiting for the upload to complete could highly motivate users to upload more pictures

misaochan commented 7 years ago

@maskaravivek I think our app already queues up pictures for upload, you can upload as many as you want without waiting for the previous uploads to complete. But waiting for WiFi could definitely help in a situation where the user has limited mobile data, then he can queue the uploads for later instead of having to remember to do it later.

neslihanturan commented 7 years ago

Do you think it is easy to implement this feature?

psh commented 6 years ago

An upload queue would be a really nice addition to the app - it would be a great debug tool for us as it gives insight (and manual control of) the sync adapter & its database.

Mockup 1 - normal operation upload queue

Mockup 2 - waiting for a wifi connection upload queue waiting for wifi

Shown in the mockups -

If we switch the retry button for an overflow menu, that would be a place to have options to Retry and Cancel a given upload. If we add a settings menu, we could also include an Advanced option to show additional low-level details about the upload and the sync adapter (if needed).

Moving the upload status to the new screen would help simplify the contributions activity / fragment. I imagine that there will be a few new settings covering whether we want to upload using mobile data, or only wifi, and if we are interested in advanced details about the uploads.

psh commented 6 years ago

I believe I made a bad assumption about how multiple share works in those mockups - looking through the code I see that although I might share 5 images in a single "batch" and they land in the database with a flag saying multiple=true there isn't a concept of a upload group ID to tie them all together. Would that be something we want to add? If so, I can create an issue for it.

nicolas-raoul commented 6 years ago

Jun 12, 2017 Google Play review:

It would have been better if there was a resume option when the data connection fails or stops as a result of poor network. Once the internet connection is lost, It has to be uploaded again. It is time consuming and also data. Please fix it. Thank you.

misaochan commented 6 years ago

@psh 's mockups for the upload queue look great to me. :)

Jun 12, 2017 Google Play review:

It would have been better if there was a resume option when the data connection fails or stops as a result of poor network. Once the internet connection is lost, It has to be uploaded again. It is time consuming and also data. Please fix it. Thank you.

I've been thinking about this, and it would be very useful indeed if we could allow the upload to be done in chunks, so that it can be paused and resumed (and also users don't lose upload progress if their internet is spotty). I am sure it CAN be done as other apps have done it, but I'm not sure how time-consuming or involved the implementation would be. @maskaravivek @psh or anyone else, any thoughts on this?

I would like to add this to our plans for next year, if it is feasible. :)

sivaraam commented 5 years ago

I would love to have the feature of pausing and resuming the ~download~ upload. I like to have it as I live in a place where the internet connection is not so good. There isn't continuously stable internet connection here. As a consequence of which there's a lot of possibility for uploads getting interrupted in between. So it would be nice to have the ability to pause downloads and resume ~downloads~ uploads.

sivaraam commented 5 years ago

Also, it would be nice if the app automatically detects a slow internet connection and switches to the limited connection mode without the user having to turn it on. And yes, the user should be allowed to turn it off if he wants to.

misaochan commented 5 years ago

Thanks for the feedback @sivaraam , happy to hear that the feature will be useful for you. :)

I am not so sure about automatically detecting a slow internet connection - what would we define as "slow"? What if the user doesn't actually want to turn on limited connectivity, but stubbornly wants to browse pictures even if his home connection is slow? He may be annoyed at having to keep turning it off.

sivaraam commented 5 years ago

I'm not sure about the feature too but just some random thoughts.

what would we define as "slow"?

Some time period within which you expect the response to have come but it hasn't?

What if the user doesn't actually want to turn on limited connectivity, but stubbornly wants to browse pictures even if his home connection is slow? He may be annoyed at having to keep turning it off.

Yeah, some features sometimes turn out to have bad consequences for a few people. I guess we could play safe by not trying to detect slow networks for the session (until app is killed) if the user turns off when we detect slow network. Also, making it customizable might also help.

I'm not going to advocate this feature a lot. Just writing out some of my thoughts about it.

neslihanturan commented 5 years ago

For slow network discussion: Maybe we can remind limited connection mode button with a silent snackbar and/or an animation on limited connection more toggle when we detect slow network, instead of forcing user.

We plan to have a limited connection mode toggle on the action bar as mentioned in #2843 . A user may want to use the app in normal mode in general (ie. using the map) but just want to wait for Wifi for uploads. Or they may want to use their connection for upload, but not for the map. For this possibility I think we can have an additional toggle as upload when wifi detected on upload steps. Ie: image

If the global "limited connection toggle" is activated, then the default value for this button should be on. If the global button is deactivated, default value for this button should be off. What do you say?

misaochan commented 5 years ago

If the global "limited connection toggle" is activated, then the default value for this button should be on. If the global button is deactivated, default value for this button should be off. What do you say?

Great idea IMO! I am all for it. :)

neslihanturan commented 5 years ago

Hi team, for upload queue UI, I loved the designs proposed by @psh : image

But they are not very similar with our existing constibutions list style: image

So here is my question, should I keep the current list design and create something similar for upload queue, or should I redesign our contributions list in a similar way with @psh 's design?

It can be good to note that we are planning to have a new overall design at the end of this year #2843 . In scope of this new design our MediaDetailsActivity will be redesigned in a similar style of our current 3 upload screens which are added by @maskaravivek . So we won't have a semi-transparent layer on MediaDetails anymore and it may be a good decision redesigning all semitransparent black components (see the layer for file names on the list). This means changing our current contributions (and explore) list style together with MediaDetailsActivity.

misaochan commented 5 years ago

I like the proposed designs, too. :) A few considerations:

images

misaochan commented 5 years ago

Edit: see https://github.com/commons-app/apps-android-commons/issues/2843#issuecomment-482805484 for more details.

misaochan commented 4 years ago

Note: When we do this we should also increase the timeout limit for network-based procedures (e.g. logging in). We currently seem to time out after about 10 seconds, based on my experience with the app when Commons was experiencing latency spikes.

misaochan commented 4 years ago

Mapbox supports offline maps if user has selected the area.

This seems like it would be useful for users with slow connections. A user is reporting that their slow connection necessitates a 1-minute download every time they load Nearby.

maskaravivek commented 4 years ago

@misaochan Can you help in defining the scope of the task from IEG point of view. I went through #2843 and I got the basic idea of how we want it to be implemented but it would be great if could define it more formally. :)

misaochan commented 4 years ago

Sure, will do that ASAP (after I manage to test Nes's PR, probably!).

misaochan commented 4 years ago

Sorry for the delayed response @maskaravivek !

From the proposal page:

Implement "limited connection" mode and allow pause/resume of uploads. Currently, the app is not very suitable to be used in locations where Wi-Fi is spotty and mobile data is limited. Losing a connection halfway through an upload means that the upload will fail, there is no way to pause and resume uploads, and no way to manage the amount of data that you want to spend on viewing your own contributions. A "limited connection mode" (that the user can toggle on and off as they wish) would automatically pause uploads until Wi-Fi is detected, and not download any images (i.e. thumbnails of previous contributions). So for instance a user with a limited mobile data plan could browse the list of Nearby places that need pictures, find one near her and take a photo of such a place, and "upload" it with a title, description and categories. The upload would be queued and would automatically resume once she uses her device in a location with Wi-Fi. Also, a user with spotty Wi-Fi connection would not need to worry if she loses connection halfway through an upload, since the upload progress would be saved. This will be useful to everyone, but especially for people in rural areas and/or the Global South where public Wi-Fi is rare and mobile data is limited and/or expensive[7]. (#746)

So there are two components to this, the ability to pause/resume uploads and the active limited connection mode.

Pause/resume uploads (this feature should be available for all uploads regardless of "limited connection" mode status):

When enabled via slider (again collaborate with @neslihanturan to place this slider in the new UI), the "limited connection mode" should:

maskaravivek commented 4 years ago

If connection is lost halfway through the upload, the upload should automatically pause (rather than fail) and should be resumed once connection is restored

When the connection is lost, the upload would fail because we will lose the HTTP connection. What we can do is to automatically retry the uploads when the user is connected to wifi. Here a fresh upload would be attempted.

So basically there should be constant connection for the duration of any single file upload. With limited connection mode we can achieve automatic queuing/retrying of uploads even if the connection is not there.

The user should have an option to pause and resume their current upload. For the UI you can work with @neslihanturan to incorporate these buttons into the v3.0 UI that she is currently working on.

Yes, right. We can provide an option to pause uploads. It will effectively cancel all ongoing uploads until the time the user chooses to resume uploads.

sivaraam commented 4 years ago

Not download any images for gallery, Explore, etc.

That sounds a little too aggressive, doesn't it? If we do that, I wonder what useful thing the user could do with the app without even browsing images. 🤔

Instead of disabling image previews may be we could queue the image download requests initiated from the "Download image" option and download them when the user connects to Wi-Fi.

As of now, when the user initiates the upload flow, we don't let him proceed further if internet is not available. With limited connection mode, we will always let him proceed and if connection is not there, we will automatically queue the upload.

I wonder why this should only be available in limited connection mode. It sounds like a great addition to the normal-connection mode, too. May be we could do this instead:

Similarly, if the upload fails due to some reason, as of now the user has to manually retry the uploads. With limited connection mode, we can automatically retry uploads based on his connectivity.

Hmm. That's not clear and in one interpretation it appears to me that it's an anti-feature for limited connection mode because you're supposed to reduce data usage. Automatically retrying seems to be the exact opposite of trying to reduce data usage. This applies regardless of the network the user is in. It would be nice if you could clarify more about this.

maskaravivek commented 4 years ago

I wonder why this should only be available in limited connection mode

Just to clarify I meant after the addition of limited connection mode feature. Yes, it should be available for all cases.

With limited connection mode, we can automatically retry uploads based on his connectivity.

Also, to clarify, based on his connectivity I mean whatever is the definition of limited connection mode. As the ticket originally mentions, if the user is on Wifi, we can automatically reattempt uploads. If he is on mobile data it won't be retried automatically.

From what I understand, the limited connection mode is to enhance app experience by allowing the user to proceed with the upload even if he doesn't have the internet. Once the upload is queued, the app should automatically upload the queued uploads when the user has connectivity. Most of the data-intensive apps do the syncing part on Wifi.

We can definitely add a setting where the user can choose when he wants the files to be uploaded, wifi, mobile data or manually.

sivaraam commented 4 years ago

Also, to clarify, based on his connectivity I mean whatever is the definition of limited connection mode. As the ticket originally mentions, if the user is on Wifi, we can automatically reattempt uploads.

Yeah, I think there's some misunderstanding between my "retry" and your "reattempt". When I said:

Automatically retrying seems to be the exact opposite of trying to reduce data usage.

I meant automatically retrying the upload whenever it fails (particularly a failure when the network is connected) at particular intervals. Looks like you were mentioning something different. Were you just mentioning the re-attempt done when the connection is restored?

Most of the data-intensive apps do the syncing part on Wifi.

The very reason why I suggested a little delay before starting our uploads. :) Using a device with limited resources, I've observed the effect first hand. The device is highly spammed and would be laggy for some time after I initially turn-on network. If newer Android versions handle this better, then this might be unnecessary.

maskaravivek commented 4 years ago

Using a device with limited resources, I've observed the effect first hand.

Right. Obviously there should be a smart retry policy. The easiest and reasonably effective way would be to have exponential backoff. But anyways retry policy is something that can be optimized iteratively. We don't need to get it right from the very beginning.

misaochan commented 4 years ago

When the connection is lost, the upload would fail because we will lose the HTTP connection. What we can do is to automatically retry the uploads when the user is connected to wifi. Here a fresh upload would be attempted.

So basically there should be constant connection for the duration of any single file upload. With limited connection mode we can achieve automatic queuing/retrying of uploads even if the connection is not there.

I was thinking more along the lines of this: https://stackoverflow.com/questions/45464795/pause-resume-httpurl-connection-while-uploading-large-files-android

It does depend on whether or not the Commons server supports resuming uploads (I don't know if it does). If it doesn't then we have no choice but to discard the upload when connection is lost and retry later. But I figured we should at least look into it?

Instead of disabling image previews may be we could queue the image download requests initiated from the "Download image" option and download them when the user connects to Wi-Fi.

Allowing users to download manually via "Download image" in the media details view is fine, I was referring to the automatic download of thumbnails that populates the Contributions etc list.

maskaravivek commented 4 years ago

Yes, as far as I could check the upload API doesn't support this header.

https://www.mediawiki.org/wiki/API:Upload

Anyways, I have opened a phab ticket to confirm the same. https://phabricator.wikimedia.org/T256359

maskaravivek commented 4 years ago

It seems that we can use stashed uploads for pause and resume. https://www.mediawiki.org/wiki/API:Upload#Example_3:_Upload_file_in_chunks

maskaravivek commented 4 years ago

@neslihanturan Can you help me with the designs for this feature.

neslihanturan commented 4 years ago

My suggestion for the toggle, and progressbar, and for contribution list. toggleRedesign

sivaraam commented 4 years ago

My suggestion for the toggle, and progressbar, and for contribution list.

That looks nice. Just a suggestion. "(Uploading… 53% of image is uploaded)" seems too verbose and the brackets seem unnecessary. I believe "Uploading… 53%" would be concise and clear. Ditto for "Paused…". Additionally, I think it would be useful to show the size corresponding to the part of the image that's left to be uploaded.

sivaraam commented 4 years ago

Allowing users to download manually via "Download image" in the media details view is fine, I was referring to the automatic download of thumbnails that populates the Contributions etc list.

Yeah. As I said before, wouldn't it be a little too aggressive, to prevent the automatic download of thumbnails that populates the Contributions etc list? Wouldn't it significantly affect the usability the app when browsing images?

misaochan commented 4 years ago

Yeah. As I said before, wouldn't it be a little too aggressive, to prevent the automatic download of thumbnails that populates the Contributions etc list?

I don't think so, as it is an option that the user can toggle on and off as they wish. Currently all users must load at least the first page of Contributions before they can even use the app. This way users who want to just, say, use Nearby to photograph a few pins while on limited data, won't have to waste their data on image downloads.

Wouldn't it significantly affect the usability the app when browsing images?

It will, but if we want maximum usability of all features all the time, there is no point to even implementing a limited connection mode, as limiting bandwidth usage is the very definition of the mode. If a user wants to browse images, they can turn limited connection mode off.

sivaraam commented 4 years ago

It will, but if we want maximum usability of all features all the time, there is no point to even implementing a limited connection mode, as limiting bandwidth usage is the very definition of the mode. If a user wants to browse images, they can turn limited connection mode off.

Yeah, that certainly would be a solution. Anyways, I was thinking if it would be nice to have multiple levels in the limited connection modes:

  1. Normal mode - App behaves the same as how it does now.
  2. Medium mode - Thumbnails images would be automatically downloaded but any upload/download would be queued until the device is connected to a Wi-Fi
  3. Aggressive mode - No thumbnails would be automatically downloaded, upload/download would be queued until the device is connected to a Wi-Fi and maybe show only the Nearby list instead of the map.

I'm not sure if its worth it, though.

misaochan commented 4 years ago

Could be a future enhancement. ;) Let's implement the basic feature first and get user feedback?

sivaraam commented 4 years ago

Let's implement the basic feature first and get user feedback?

Sure. :)

maskaravivek commented 4 years ago

Here are the changes that I am making for the remaining pieces of this feature:

  1. Limited Connection mode toggler in the toolbar
  2. Enabling Limited Connection mode would not upload image even when online
  3. As soon as the user disables the Limited Connection mode sync will start to sync all images which the user tried to upload while in offline mode 4 Upload failures in Limited Connection mode will be handled via the usual flow as in they will be shown in the contribution list with failed-retry option

cc @misaochan

sivaraam commented 4 years ago

Offline mode toggler in the toolbar

So, "limited-connection" mode had turned to "offline" mode now. It's a little weird because I see our app's primary purpose as uploading images and then to browse them. So, a "Offline" mode sounds contradictory right in it's name. I didn't get this feeling for "limited-connection" mode. We should like think about the name 🤔

When in offline mode, Category Selection Fragment won't be shown, and any image processing which requires internet will be skipped

This sounds a little off to me. So if a user enables offline mode, they would have no way to add categories to the image they upload via the app? I wonder if that's going too much to conserve data as it's clearly curbing such users from a useful feature of the app. I would say it's better to allow users to add categories to the uploaded image regardless of the mode.

Ditto for the image processing checks. I feel that there shouldn't be a problem with doing any check that does not upload the whole picture. I just don't get why we should skip significant /useful checks just because a user has limited data. For instance, wouldn't it be better if we do the "Image already present in Commons" check and warn the user before hand about that instead of skipping that check; allowing user to upload it; have that image possibly deleted as a duplicate later?

Correct me if I got something wrong.

maskaravivek commented 4 years ago

@sivaraam Sorry, for the confusion. I have updated the scope based on your comment.

sivaraam commented 4 years ago

Nice. I just have one doubt for now:

Enabling Limited Connection mode would not upload image even when online

Does this apply only for mobile data or also for Wi-Fi? I'm mostly just trying to get a refined meaning for "online" here. My understanding is that this should happen only for mobile-data per https://github.com/commons-app/apps-android-commons/issues/746#issuecomment-647369932:

A "limited connection mode" (that the user can toggle on and off as they wish) would automatically pause uploads until Wi-Fi is detected, and not download any images (i.e. thumbnails of previous contributions).

maskaravivek commented 4 years ago

For now, limited connection mode if enabled would be effective on any network the user might be using. In the future, we can have another set in the Settings page where he can define his preference.

misaochan commented 4 years ago

Here are the changes that I am making for the remaining pieces of this feature:

  1. Limited Connection mode toggler in the toolbar
  2. Enabling Limited Connection mode would not upload image even when online
  3. As soon as the user disables the Limited Connection mode sync will start to sync all images which the user tried to upload while in offline mode 4 Upload failures in Limited Connection mode will be handled via the usual flow as in they will be shown in the contribution list with failed-retry option

Makes sense to me @maskaravivek .

Does this apply only for mobile data or also for Wi-Fi? I'm mostly just trying to get a refined meaning for "online" here.

Hmm, good point @sivaraam . I should have defined that more clearly in the scope. At the moment it feels like adding Wi-Fi detection would be adding a lot of complexity to the implementation. After the basic feature is out, perhaps we can get feedback from users on their preference for whether or not they'd like network detection.