Upload photo information

[bsaiken]

@damirius You did not aNswer my question from a different Issue, but this would likely be a new Issue anyway.

Is it possible, when we are uploading a photo, to upload the photo information: timestamp and the file name (as it appears on the mobile device)?

[damirius]

@bsaiken

Sorry about that. Yes it is possible, there's event that is being triggered when we start uploading each image, we already use it for some other stuff. We can make another ajax call there to store that info. What's the purpose of that and where we should store that stuff?

[bsaiken]

@damirius First of all, we should be storing the timestamp anyway. I will make a new Issue for that.

But, more importantly, this is my big idea! Let's store the file name first. Then, if we get a timeout when the FT is in the field, it should be possible for the FT to initiate an independent upload of the all the photos once they have a WiFi connection. This upload sequence would not be subject to any timeout. We just match the photo to the stored file name.

Tell me if I am correct.

And one more question. Isn't there a comment in dropzone to upload thumbnails?

[damirius]

@bsaiken

Thumbnails are shown in the dropzone field, they aren't actually uploaded on the server. Resize option is popular one https://github.com/enyo/dropzone/issues/781, but they still didn't add it in the dropzone. As far as I can see most of the people tried some other solutions or some little hacks to change the code of the dropzone so that it uploads resized images.

[FWAJL]

@damirius @bsaiken

Read http://www.dropzonejs.com/#enqueuing-file-uploads to queue file upload until the ft clicks the start upload:

If you set autoProcessQueue to false, then .processQueue() is never called implicitly. This means that you have to call it yourself when you want to upload all files currently queued.

See if we can retrieve the queue in JSON format, save it to the db and reuse it later for upload

Here are the steps we think we need to take to validate this possible solution: 1- set autoProcessQueue to false on DropZone 2- upload a few files anywhere. 3- retrieve the queue as JSON if possible. 4- save it in cookie or local storage and try to launch a new dropzone "instance" to upload the saved queue.

If those 4 steps are successful, it means we can store the JSON to the database and then later, we retrieve the JSON and repeat step 4 in the situation we want.

Can you try that please @damirius ?

[bsaiken]

@damirius I don't know if this is helpful or not, but in case...

I dug into dropzone.js. I put a break at line 498 and a watch on the variable file. See attached photo. It seems like that's where the information we need is.

[damirius]

@bsaiken @FWAJL

As I said dropzone is firing few public accessible events on file drop and on file send which can provide us with file variable. We can queue files and try solution that Jeremie suggested. The question is where will we add that separate window for uploading queued files when FT gets his WiFi connection? Do we need separate window at all or we just need to add button in the infowindow or we will add new page just for that.

[bsaiken]

@damirius You seem confident this will work. Jeremie and I not so much, so we are taking this one step at a time.

The "Upload Photos" button could go a number of places, and I haven't decided for certain yet. It seemed moot until we knew whether or not it worked. One option is to include a "Wifi Photo Upload" item on the main menu. Another option is to automatically add a checklist item as soon as a queue is initiated. For now, it really does not matter - you could make a dummy page - whatever is easiest for this development stage.

[damirius]

@bsaiken @FWAJL

Yap, that thing with queue doesn't work. Those file objects are used just for the flow control of the process.

[FWAJL]

@damirius @bsaiken so we can't save the files in queue and re-inject them later?

[damirius]

@FWAJL @bsaiken

We can save those "files" objects but they are there just for the control of the upload process, they contain some stuff but we can't inject them later to try and upload the actual files.

[FWAJL]

@damirius @bsaiken Yes, I'm not seeing either the actual info that would link the object file to the physical file itself after. At least for now.

[FWAJL]

@damirius @bsaiken

More on the subject:

Since it is not possible to retrieve the data from Dropzone, I'm investigating a possible solution to store the documents (pictures and pdf) on the FT's device. I'm not sure where yet but I have an idea. Give me a little bit of time to build the case.

The goal would be to use Dropzone to store the data on the FT's device and then to the server when the connection is WIFI.

Thank you for your help so far :+1:

[FWAJL]

@bsaiken @damirius

So, I have found a possible solution to test out on a dummy to start with. I found a clear enough example (http://blog.jankuca.com/post/7169506266/offline-file-uploads-file-api) with the source code (https://github.com/jankuca/offline-uploader). Alternatively, we could use this example: https://github.com/agektmr/BrowserStorageAbuser

Basically, we would use IndexedDb (first choice if available) and WebSQL storage to store the documents on the device. There are quota to respect depending on the platform (iOS or Android if we limit ourselves to those). To find out the limit: go here on your mobile devices here and try to fill the storage with 10 chunks of 5mb (50mb) which I think is a large amount of data. On my Samsung S4 mini (Android 4.4.2), I can store 50mb in IndexedDB storage and WebSQL. _I tried add the chunk offline and online and it works._ The only thing is and it may be my phone, the chunk creation is resource heavy

To optimise the storage, I think we should resize images down since recent smartphone can take pictures of several MB. @bsaiken : what would be an acceptable image size in pixels?

I don't know if we can use Dropzone or not. To be confirmed but I think we must try this as it could be the first step to make the whole app offline.

What do you think?

Some more reading and resources: http://www.html5rocks.com/en/tutorials/offline/quota-research/

[bsaiken]

@FWAJL That's not enough data. I will have to look for another solution.

[FWAJL]

@bsaiken @damirius @gsouvik

Please definieWhat are your requirements:

• how much do you think you need to store?
• have you tried this link http://demo.agektmr.com/storage/ on your iPad and iPhone? Play with it using chunks of 500kb for example (a picture size big enough to record field parameters for example).
• can you answer this question: what would be an acceptable image size in pixels?

I have tried 100 chunks of 500kb (~50MB) with IndexedDB (max. on android 4.4.2) and it still works, online and offline. Then I have tried 200 chunks of 500kb (~100MB) with WebSQL and it worked up to 74.8MB because my phone limited space available (done online and offline).

That represents between 100 and 200 documents in total!

We would need to test on lower Android versions (I can test 4.1.2 and 4.4.4) and of iOS. Then we could set a min. platform version that we support because like Souvik said, support is limited but platform updates happens often on recent smartphones.

If you dismiss this solution from my first comment, I only see the native apps as the real solution. These HTML5 technologies are the best chances to stick to the Web App.

Thank you.

[bsaiken]

@FWAJL I am not in control of the data size requirements. Photos on my phone are ~ 2Mb. Some phones have larger sized images maybe 8Mb or more. I can't really predict how many photos a FT will need to upload. I was thinking 50Mb might be too small. However, if we can limit the file size to 1Mb or less, and we had 50Mb to work with, that would probably be OK. Can you mock this up for me? The only way to really test this is on my mobile phone in a real situation.

[FWAJL]

@bsaiken

I understand. The reason why I'm talking about resizing pictures is because we don't need to full HD pictures to record a field parameter.

However, can you at least provide some start-of-answer to the questions I posed, please:

• how much do you think we need to store based on your experience of field work?
• have you tried this link http://demo.agektmr.com/storage/ on your iPad and iPhone? Play with it using chunks of 500kb for example (a picture size big enough to record field parameters for example).
• can you answer this question: what would be an acceptable image size in pixels?

These questions are to be used to reflect on the need. I'm not asking how many pictures of FT will upload. We will see first how many _we_ can store to decide if this lead can be implement as a viable solution.

I will try to make you a mockup but it will probably be outside the repository context.

Thank you.

[bsaiken]

@WebDevJL I answered some of these questions. Here is a more comprehensive answer.

how much do you think we need to store based on your experience of field work?

I don't know. We need a flexible solution. Above, I suggested we use 50Mb. However, this number is a product of file size and # of photos, so it is not that important from a needs point of view.

have you tried this link http://demo.agektmr.com/storage/ on your iPad and iPhone? Play with it using chunks of 500kb for example (a picture size big enough to record field parameters for example).

Yes. Not happy with the result.

can you answer this question: what would be an acceptable image size in pixels?

Above, I stated 1Mb. However, the photo I just posted, which has plenty of resolution for our work, is 85kb. Perhaps the solution we are seeking just to upload much smaller image sizes. I can't give you a hard number here, either. Most image needs would be small, but in some cases high resolution images would be advantageous.

I am investigating other options since I don't like the direction we are headed. I suggest we all focus on other Issues until I can gather more information.

[bsaiken]

@WebDevJL Can you please check this out and see if it is a possible solution to queue caching: http://docs.fineuploader.com/branch/master/features/pause.html

[WebDevJL]

@bsaiken

This software is not free therefore, we need to purchase it to try ($69/year). The documentation talks about a specific case so without trying, I can't tell if it is a possible solution.

Thank you.

[bsaiken]

I could Skype tomorrow if you are free.

Brian

On 2015-08-30 11:13, Jérémie Litzler wrote:

@bsaiken [1]

This software is not free therefore, we need to purchase it to try ($69/year). The documentation talks about a specific case so without trying, I can't tell if it is a possible solution.

Thank you.

Reply to this email directly or view it on GitHub [2].

*

Links:

[1] https://github.com/bsaiken [2] https://github.com/FWAJL/FieldWorkManager/issues/1250#issuecomment-136156573

[bsaiken]

@WebDevJL Jeremie, I got a response from fine-uploader. They say it's easy. Please take a look at their response.
http://stackoverflow.com/questions/32263174/fine-uploader-delayed-upload-queue

[WebDevJL]

@bsaiken I have posted a comment to see if we can really do what we want.

[bsaiken]

@WebDevJL WHere did you post your comment - I see no comment.

[FWAJL]

@bsaiken Sorry, I meant on stackoverflow, not here. It was deleted by an admin.

Can you ask this to Ray:

Ray,

I want to add to Brian's questions: what happens if the user closes the browser or switch off the phone? Will the postponed upload by clicking that Upload button still work or do we have to reselect the files ? Where is stored the info about the uploading file?

The use case described by Brian is to only initiate upload in a specific task and then halt the upload until the user has access to a WIFI connection. When connected to the WIFI, the user can click that button to upload the files. I saw that you have a pause functionality. Can it be used in that case?

Thank you.

[bsaiken]

I understand that. I do not see your comment on Stack Overflow.

Brian

On 2015-09-03 15:06, FWAJL wrote:

@bsaiken [1] Sorry, I meant on stackoverflow, not here.

Reply to this email directly or view it on GitHub [2].

*

Links:

[1] https://github.com/bsaiken [2] https://github.com/FWAJL/FieldWorkManager/issues/1250#issuecomment-137559950

[FWAJL]

@bsaiken that's because it was deleted by an admin of stackoverflow.

Please reply yourself with this:

Ray,

I want to add to Brian's questions: what happens if the user closes the browser or switch off the phone? Will the postponed upload by clicking that Upload button still work or do we have to reselect the files ? Where is stored the info about the uploading file?

The use case described by Brian is to only initiate upload in a specific task and then halt the upload until the user has access to a WIFI connection. When connected to the WIFI, the user can click that button to upload the files. I saw that you have a pause functionality. Can it be used in that case?

Thank you.

[bsaiken]

@WebDevJL I can see why they deleted your comment. Your questions are a no-no on Stack Overflow. I'm actually surprised they left mine.

The questions you raise are sort of doomsday scenarios that could happen to any technology. If I collect data on paper, the data sheets could blow out the door of my truck - and I don't want to admit how many times that has happened.

If all we need to do to be successful is to keep the browser open and not turn off the phone it's not a fatal flaw - most people do that anyway. Almost everyone either goes home or to a hotel with WiFi at the end of the day so it could just be part of the day's work.

I am going to buy the license and we'll try it out. Souvik is down to low and very low priority items, so I am planning to assign it to him unless you feel strongly about doing it yourself.

[bsaiken]

@WebDevJL Actually, we don't need to purchase it until we know it will work for us, so it's really risk free. In fact, we would not need to buy a license unless we started making money - it's free to develop, just like Google Maps.

[WebDevJL]

@bsaiken @gsouvik

Souvik can start working on it, no problem. I'm on #1270 right now.

[bsaiken]

@gsouvik @WebDevJL Jeremie, You may want to merge the DEV branch to feature_devsupport because I uploaded the fine_uploader archive to the DEV branch.

Souvik, This has been a real challenge for us to implement. Let me know if you need help understanding the requirement. We could have a Skype chat if needed. The first step would be to implement a simple upload (1 photo only) from the mobile site. I will create a new Milestone with Issues for each step.

[FWAJL]

@bsaiken @gsouvik merge complete.

[rnicholus]

I'm the maintainer of Fine Uploader. Can I be of assistance here in any way as far as answering further questions?

[bsaiken]

@rnicholus I bought a Fine Uploader license and I have tech to implement the code. Good to know who to ask if we have questions, though. Thanks.

[rnicholus]

Generally speaking, we like "how do I do this" type of questions to be posted on SO under the fine-uploader tag (which I monitor regularly), and feature/bug reports/requests in our issue tracker. But I'd be happy to provide guidance in this thread for your implementation effort.

[bsaiken]

@rnicholus @gsouvik @FWAJL Sure, if we have a question this post will notify everyone to ask questions under Fine Uploader's git.
Jeremie may have a follow-up question from my original Stack Overflow question (http://stackoverflow.com/questions/32263174/fine-uploader-delayed-upload-queue).

[rnicholus]

If the questions are very specific to your implementation, you can simply ask in this thread, as I am now watching.

[bsaiken]

from #1282.

@gsouvik said: technicians just queue up the files and keep the page/modal as it is?

No. This would not help. The technician will need to be able to navigate to other pages and do other work while adding to the upload queue.

@gsouvik said: The fine-uploader version...doesn't use jQuery.

I just downloaded the wrong library. Please just download the correct library.

[rnicholus]

Note that there is no real advantage to pulling in jQuery-wrapped Fine Uploader. See FineUploader/fine-uploader#1310 for more details.

[gsouvik]

@rnicholus Thanks for sharing that thread on jQuery wrapper, it really helped. I need a bit of more information regarding a different area of the library. Is it possible to store up the queue itself and upload it at a later time? I'll try to explain that with an example. Say I have a page where I've configured fineuploader with manual upload trigger. I add some files to upload, but don't start the upload. I navigate away to some other page or close the browser. Is it possible for me to get back the entire queue when I open this page again, so that I may start the upload. The idea is, I'm in a situation where I need to create the queue but must be able to upload it at a later time without re-selecting the files. Thank you.

[rnicholus]

I navigate away to some other page or close the browser. Is it possible for me to get back the entire queue when I open this page again, so that I may start the upload.

No. To support this, one of the following would have to be happen:

1. Each file would have to be stored in the browser. One option is localStorage but space in that store is limited. localStorage is meant for small amounts of text, such as JavaScript objects. Simple key/value store. Another option would be to store the files in IndexedDB. This is possible, but not currently supported by Fine Uploader. Even if we did support this, it would be limited to very new browsers (nothing older than IE10) and managing this type of data in IndexedDB would likely add quite a bit more complexity to Fine Uploader.
2. Another alternative would be to store the paths to the files on the user's local filesystem. In theory, this seems simple, but there is no way to gain unrestricted access to the user's filesystem, for obvious security reasons. The closest thing to this type of access is via a sandboxed filesystem provided by the Filesystem API specification. Currently, that W3C spec is deprecated, and only Chrome implements it.

[bsaiken]

@gsouvik @rnicholus One of the advantages of not being as tech savvy as the rest of you is that I'm too stupid enough to realize it won't work. I see three possibilities where this could still work - correct me if I am wrong.

1) Specify the exclusive use of Chrome. The only disadvantage is one might expect Filesystem API would eventually be deprecated in Chrome as well. 2) Store the files in IndexedDB. I understand this is not supported by fine-uploader - what level of effort would it take to code a fine-uploader plug-in that would make it work on our site? 3) Since the limitation arises when we close the browser and/or navigate away from the page, how about this: First - assume the user would not close the browser until the photos in the queue are uploaded. Second, could we create a separate upload page that stays open as a separate tab or separate window? Each time the user selects a new file to upload, they are directed to the "upload" page, where the queue is staged. On the upload page there are buttons to either a) return to the origination page (in the other window or tab) or b) upload the queued photos.

[rnicholus]

Specify the exclusive use of Chrome. The only disadvantage is one might expect Filesystem API would eventually be deprecated in Chrome as well.

We are already making use of the Filesystem API to accept folder uploads via DnD and the <input type="file"> element. But since this is a dead specification, the implementation in Chrome/Opera is now considered proprietary/non-standard. Even though Google will likely keep this in Blink until a suitable replacement arrives in the form of a new standard, it's probably not wise to code against this any further. As such, we will not invest any additional time/resources building upon this implementation.

Store the files in IndexedDB. ...what level of effort would it take to code a fine-uploader plug-in that would make it work on our site?

I'm not sure at the moment, as this is not a web specification that I have much practical experience with. I am fairly certain that this will not be a trivial feature to implement, and I am hesitant to code up a feature as part of the library that can potentially persist large amounts of data to a user's filesystem. This feature could not be unbounded. Complexity follows. However, you could certainly code this up on your end. Simply grab each file from Fine Uploader's API as it is submitted via the getFile method, store it in IndexedDB, and them remove it once the file has been uploaded. When an instance is created, check IndexedDB for files. If any exist, feed them into Fine Uploader via the addFiles API method.

[bsaiken]

@rnicholus @gsouvik Ray, I am not expecting any code modifications to fine-uploader on your part, my query to you is only to know if it can be done and to verify that any modifications we make for incorporation into our website are not an infringement on code that is rightfully yours. If it does turn out we are able to make modifications, and you like them, you would be welcome to it.

The question of the level of effort required is directed to Souvik, who would take on the work. If the coding is too complex (a.k.a. expensive) we may need to stand down - I'm a one man shop with limited resources and guile can only get me so far.

[rnicholus]

The concept is straightforward, but the devil is always in the details. Though this should be a possible integration given Fine Uploader's comprehensive API.

Your biggest obstacle may be browser support. First, only Firefox & Chrome/Opera fully support IndexedDB. IE10+ and Safari for OSX only have partial support for the spec. Also, in typical Apple form, the implementation of IndexedDB was horribly botched in iOS8. They haven't fixed it in iOS9 either as far as I know.

I honestly can't give you an accurate estimate as to how much work this will be as I am not very familiar with the details of this API as I have not used it before for anything non-trivial. There is an appropriate example that you will likely benefit from at https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/.

All said, this doesn't seem like a particularly difficult API to work with (famous last words). Also, I think this will be an interesting integration. If Fine Uploader work ever dies down a bit, I may consider writing a blog post detailing this type of workflow support.

[bsaiken]

@rnicholus wrote: Your biggest obstacle may be browser support.

This is not an obstacle for me since I can specify a browser to my users (if necessary). This is no more restrictive than requiring users to access the site with a custom mobile app.

@gsouvik - Souvik, when you consider how we might implement this, I believe our best option is to use Chrome. All my testing will be using Chrome. We only need the mobile UI to work on the mobile version of one browser.

[rnicholus]

Keep in mind that Chrome on iOS is, for the most part, just a less performant version of Safari, with less features. It does not share the same feature set or codebase as its desktop cousin. On Wed, Sep 16, 2015 at 12:20 AM bsaiken notifications@github.com wrote:

@rnicholus https://github.com/rnicholus wrote: Your biggest obstacle may be browser support.

This is not an obstacle for me since I can specify a browser to my users (if necessary). This is no more restrictive than requiring users to access the site with a custom mobile app.

@gsouvik https://github.com/gsouvik - Souvik, when you consider how we might implement this, I believe our best option is to use Chrome. All my testing will be using Chrome. We only need the mobile UI to work on the mobile version of one browser.

— Reply to this email directly or view it on GitHub https://github.com/FWAJL/FieldWorkManager/issues/1250#issuecomment-140628794 .

[gsouvik]

@bsaiken After going through the discussion thread, from my view point, the following two options seems to be somewhat feasible: 1) As you suggested, using a separate upload page and keep adding everything to it. The user must not close the tab (the page loaded in it) or else they loose the queue. They must be educated regarding this.

2) IndexedDB - I haven't worked with it before, but can research and see how much it might work. I think @WebDevJL already did some research on it and has some data, we may use those, though I may be wrong.

Either way, to store the queue is the biggest challenge and I think we need to do some experiments before we could actually adopt one of the options. Please share your thought, which one you would want me to try.

[WebDevJL]

@bsaiken @gsouvik

Yes, you're right, Souvik. Look at this comment for details about IndexedDb / WebSQL implementation example: https://github.com/FWAJL/FieldWorkManager/issues/1250#issuecomment-135111166

[bsaiken]

@WebDevJL Jeremie, your opinion about the upload page?