search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.59k stars 10.59k forks source link

File upload with Webdav #11666

Open Adam212 opened 6 years ago

Adam212 commented 6 years ago

Hello,

we're using upload over WebDAV to Nextcloud with encryption. When sending bigger files (more than 20MB) it's getting an error 500, because checking if file exists after upload is before upload has finished, see logs below:

10.120.104.1 - rocket [02/Aug/2018:10:05:05 +0200] "GET /remote.php/webdav/AzXTGobezzwM968XN HTTP/1.1" 404 5 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
10.120.104.1 - rocket [02/Aug/2018:10:05:06 +0200] "DELETE /remote.php/webdav/AzXTGobezzwM968XN HTTP/1.1" 404 5 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
10.120.104.1 - rocket [02/Aug/2018:10:05:06 +0200] "DELETE /remote.php/webdav/AzXTGobezzwM968XN HTTP/1.1" 404 5 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
10.120.104.1 - rocket [02/Aug/2018:10:05:06 +0200] "PUT /remote.php/webdav/AzXTGobezzwM968XN HTTP/1.1" 201 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
10.120.104.1 - rocket [02/Aug/2018:10:05:06 +0200] "DELETE /remote.php/webdav/AzXTGobezzwM968XN HTTP/1.1" 204 0 "-" "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"
karakayasemi commented 6 years ago

Hey @Adam212, thank you for reporting. It is a known issue of UploadFS package. In here https://github.com/RocketChat/Rocket.Chat/blob/develop/packages/rocketchat-file-upload/ufs/Webdav/server.js#L113 we have a "to do" for this problem. You can extend setTimeout value from 500 to more higher value as a temporary solution. @ggazzo FYI.

oliinykdm commented 6 years ago

Hi @karakayasemi, how can I fix this issue manually without increasing the "setTimeout" value because this does not solve the issue at all, for small files it takes a lot of time, but for the large file is also not helped.

karakayasemi commented 6 years ago

Hi @oliynykdm, I spent lots of time on it, to somehow hack this bug on Rocket.Chat side, but I could not. The problem is UploadFS store trying to read the written file in the on finish event of write stream. However, on finish event is emitting before the writing process is not finished on the WebDAV server. So UploadFS store cannot read the file and fail. IMHO, we should try to fix this problem inside of this callback on ufs-store: https://github.com/jalik/jalik-ufs/blob/master/ufs-store.js#L250 . Maybe @jalik can show a way to us about this topic.

oliinykdm commented 6 years ago

Hi @karakayasemi. Thank you for your reply. I've temporarily solved this issue by using mounted SFTP in the system, so the Rocket.Chat detects it as a simple dir in the system.

karakayasemi commented 6 years ago

@oliynykdm So, you are not using WebDAV store any more right?

oliinykdm commented 6 years ago

@karakayasemi Yes, while WebDAV cannot be workable, I will use SFTP. My storage hosting provider allows using any protocols, SFTP, FTP, WebDAV. I'll use WebDAV once it'll be usable, the SFTP solution is a little bit handcrafted.

didacroyo commented 4 years ago

Hi guys! I'm the only one missing the option to "share link" to the target file, instead of only the option to "upload" the file to rocketchat? in my view, one of the key benefits of the integration with NextCloud is preventing to have large and duplicate files that already exist in the NextCloud storage. image

gautamilango commented 4 years ago

Hi guys! I'm the only one missing the option to "share link" to the target file, instead of only the option to "upload" the file to rocketchat? in my view, one of the key benefits of the integration with NextCloud is preventing to have large and duplicate files that already exist in the NextCloud storage. image

Is there a way to just share a link to that publicly shared file on NextCloud instead of having to reupload the file over RocketChat? If not, we can open a feature request.

jalik commented 4 years ago

Hi @oliynykdm, I spent lots of time on it, to somehow hack this bug on Rocket.Chat side, but I could not. The problem is UploadFS store trying to read the written file in the on finish event of write stream. However, on finish event is emitting before the writing process is not finished on the WebDAV server. So UploadFS store cannot read the file and fail. IMHO, we should try to fix this problem inside of this callback on ufs-store: https://github.com/jalik/jalik-ufs/blob/master/ufs-store.js#L250 . Maybe @jalik can show a way to us about this topic.

It seems that I came too late, and I don't know how I missed this topic, but did you managed to workaround the issue ?

As I understand, you are talking about this LoC https://github.com/jalik/meteor-jalik-ufs/blob/master/ufs-store.js#L247 which is required to finalize uploaded (defining size, etag, token, url...) ? how could this problem be solved ? (I don't know how webdav works and why it is incompatible with the current behavior whereas local storage is working fine for example)

Mystikal57 commented 4 years ago

Hi guys! I'm the only one missing the option to "share link" to the target file, instead of only the option to "upload" the file to rocketchat? in my view, one of the key benefits of the integration with NextCloud is preventing to have large and duplicate files that already exist in the NextCloud storage. image

We are also interested to have a share link feature ! It's a non sense to upload the file to rocketchat