Flowjs

[sorenwacker]

Hi,

could you have a look at this?

The multi file upload is working as well as the progressbar.

The last thing that I am struggling with is to return the uploaded filenames back to the du callback.

The code contains a lot of log messages at this point.

[sorenwacker]

I added a minimal demo application in `demo/demo.py

the usage.py will create an error.

[sorenwacker]

Now 'usage.py' also works as expected. Yippy!!

[sorenwacker]

I though it is not working, but it actually does. I was trying to upload a file with wrong suffix. There should be a message, this file name not accepted... So, multiple files can be uploaded and the progressbar is smooth. Now, I will try to show the cancel button.

[fohrloop]

Hi @sorenwacker,

Thanks for the PR! I took a look at your code. Would you like me to merge some changes? If so, I would suggest to create smaller commits with comments so it would be easier to follow what commits are for what purpose. I think one PR per one topic could be easier, so I can merge fast.

There are also tests in the .\tests folder. If you are changing or adding functionality, please also include some test for it in there. The tests are ran with python -m pytest.

[sorenwacker]

Hi, I was not sure if I can fix it. So, my first attempt was to get it working. Now, I can create PRs more frequently. Though, I am pretty new to React, lots of what I am doing is trial and error.

Do you have any suggestions what might be missing to make the Cancel button appear when the upload is active?

[sorenwacker]

You can already merge these changes if you want. At least the upload is working now with flowjs and also multi file upload works.

[sorenwacker]

This pull request enables the Pause/Resume and Cancel button.

[sorenwacker]

Cancel and Pause/Resume button work.

[sorenwacker]

Any updates on this?

[fohrloop]

Hi! I Have actually been doing some updates on the JS libraries dash-uploader lately and have had this PR on my mind. Some day when I have some time I will take a closer look on this. It would be nice to replace resumable.js with flows.js and get the multiupload working!

[sorenwacker]

It’s working for me. Only a small bug left. The cancel button disappears after the first file. Should be easy to fix.

---

From: Niko Pasanen @.> Sent: Thursday, December 9, 2021 3:41:53 PM To: np-8/dash-uploader @.> Cc: Sören Wacker @.>; Mention @.> Subject: Re: [np-8/dash-uploader] Flowjs (#50)

Hi! I Have actually been doing some updates on the JS libraries dash-uploader lately and have had this PR on my mind. Some day when I have some time I will take a closer look on this. It would be nice to replace resumable.js with flows.js and get the multiupload working!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/np-8/dash-uploader/pull/50#issuecomment-990365909, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAZ4A7RGIEJKJOEJOR3YBVTUQEWDDANCNFSM5EDVKK2A. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[fohrloop]

Hi! I'm now trying to merge the changes you have done to the development branch. First problem in the merge is that there has been some changes in the dev branch and I first try to get the flowjs branch updated with all the newest changes, including all JS updates etc. The second problem that I'm facing is that I want to keep the commit history very tidy, all commit messages clear and commits having ideally just one change each. Even if flowjs branch would be up to date, the commits in this PR will need some heavy refactoring.

I have tried this branch (the commit 8e50e1a), and could install it with the --legacy-peer-deps flag. I also managed in simple tests to use the component to upload multiple .csv files. Hooray! However, it still needs some fixes. I noticed that having files of wrong filetype will not work as expected (was expecting a error popup). Also, the multiupload functionality had some random errors. As an example, I had 5 csv files, (3 x 1.7Mb, 2 x 233Mb), and got the callback to fire on the small ones but not on the large ones. All files seem to be uploaded to the upload folder, though. See:

Here are the errors in the browser console:

And here some errors from the console:

Click to expand!

``` Traceback (most recent call last): File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 42, in post return self._post() File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 113, in _post os.unlink(target_file_name) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Niko\\AppData\\Local\\Temp\\Uploads\\99650dea-64f4-11ec-a002-34cff6f30b32\\some_large_data1.csv' Traceback (most recent call last): File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 42, in post return self._post() File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 113, in _post os.unlink(target_file_name) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Niko\\AppData\\Local\\Temp\\Uploads\\99650dea-64f4-11ec-a002-34cff6f30b32\\some_large_data1.csv' Traceback (most recent call last): File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2091, in __call__ return self.wsgi_app(environ, start_response) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1519, in full_dispatch_request return self.finalize_request(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1538, in finalize_request response = self.make_response(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1701, in make_response raise TypeError( TypeError: The view function for 'post' did not return a valid response. The function either returned None or ended without a return statement. Traceback (most recent call last): File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2091, in __call__ return self.wsgi_app(environ, start_response) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1519, in full_dispatch_request return self.finalize_request(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1538, in finalize_request response = self.make_response(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1701, in make_response raise TypeError( TypeError: The view function for 'post' did not return a valid response. The function either returned None or ended without a return statement. [2021-12-24 22:03:59,072] DEBUG in httprequesthandler: File saved to: C:\Users\Niko\AppData\Local\Temp\Uploads\99650dea-64f4-11ec-a002-34cff6f30b32\some_large_data1.csv Traceback (most recent call last): File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 42, in post return self._post() File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 113, in _post os.unlink(target_file_name) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Niko\\AppData\\Local\\Temp\\Uploads\\99650dea-64f4-11ec-a002-34cff6f30b32\\some_large_data2.csv' Traceback (most recent call last): File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 42, in post return self._post() File "C:\Users\Niko\dash_uploader\dash_uploader\httprequesthandler.py", line 113, in _post os.unlink(target_file_name) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Niko\\AppData\\Local\\Temp\\Uploads\\99650dea-64f4-11ec-a002-34cff6f30b32\\some_large_data2.csv' Traceback (most recent call last): File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2091, in __call__ return self.wsgi_app(environ, start_response) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1519, in full_dispatch_request return self.finalize_request(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1538, in finalize_request response = self.make_response(rv) File "C:\Python\venvs\dash-uploader\Lib\site-packages\flask\app.py", line 1701, in make_response raise TypeError( TypeError: The view function for 'post' did not return a valid response. The function either returned None or ended without a return statement. [2021-12-24 22:04:00,155] DEBUG in httprequesthandler: File saved to: C:\Users\Niko\AppData\Local\Temp\Uploads\99650dea-64f4-11ec-a002-34cff6f30b32\somedata.csv [2021-12-24 22:04:00,161] DEBUG in httprequesthandler: File saved to: C:\Users\Niko\AppData\Local\Temp\Uploads\99650dea-64f4-11ec-a002-34cff6f30b32\somedata2.csv [2021-12-24 22:04:00,170] DEBUG in httprequesthandler: File saved to: C:\Users\Niko\AppData\Local\Temp\Uploads\99650dea-64f4-11ec-a002-34cff6f30b32\somedata3.csv 3 ['somedata.csv', 'somedata2.csv', 'somedata3.csv'] 99650dea-64f4-11ec-a002-34cff6f30b32 False somedata3.csv [2021-12-24 22:04:02,220] DEBUG in httprequesthandler: File saved to: C:\Users\Niko\AppData\Local\Temp\Uploads\99650dea-64f4-11ec-a002-34cff6f30b32\some_large_data2.csv ```

I would guess that the server-side logic would just need some more retries when handling the uploaded files. I have to make sure that proper tests are added to test this!

What I'm going to do next is to take copy changes from your commits to the flow-dev branch where I have the flowjs and newest dev merged! I'll keep you updated.

[sorenwacker]

I sorry for the messy commits. I am not used to work on other peoples projects. I am not sure how to refactor a past commit. I wonder if the tests got implemented in the mean time in this or another branch. It is also the first time I am working with React. I appreciate the feedback though. It is always good to get that, to know where to improve.

[sorenwacker]

I fixed the disappearing buttons, Cancel and Pause/Resume will be visible until the last file is finished. And I changed the final message to '... 4 files'. The name of the last file is better than just showing it. I tried to reproduce the error that you had, but couldn't. I guess there is some string generated wrongly. But the upload seems to work fine even for larger files (~1GB).

[sorenwacker]

It would be nice to also have a filter for files which are already on the server.

[fohrloop]

Hi @sorenwacker,

I have been working on and off with this for a while now. Your commits in this PR are already taken into account in the flow-dev branch I am working on. It is currently the most-up-to-date branch there and I will merge that to dev at the point when I get the basic functionality working, as many things are getting broken with the flow.js update.

Sorry for the inconvenience, and the dev experience currently is not the best as there are many branches, some of them stale and should be removed, and the flow-dev branch has so many things going on and nothing is working :D I'll try to push myself to fix the errors and update the dev branch. I'm also working to fix all the tests.

I have to think about the "filter for files" functionality. You can also open an issue for it if you think that is applicable. I also opened Discussions board here, so you are welcomed also to share your ideas there :)

[fohrloop]

The flow-dev was merged into dev in 511557f587a02fbae6a0462e5074abd449c14bd0. This contains also the changes in the commits in this PR. Thanks for your help. Soon there will be dash-uploader 0.7.0 with many things fixed or improved! 🎉

[sorenwacker]

Welcome. Is the new pre-release already based on flowjs?

[fohrloop]

Yes it is