search

crossbario / crossbar

Crossbar.io - WAMP application router
https://crossbar.io/
Other
2.05k stars 274 forks source link

File Upload Service Python 3 - Using code from Docs #481

Closed singlecheeze closed 8 years ago

singlecheeze commented 8 years ago

Using code from docs: https://github.com/crossbario/crossbarexamples/tree/master/fileupload

Also please see my comment on Need callback for File Upload Service #466 about docs discrepancy...

2015-10-14T17:17:25-0400 [Router       8792] Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/log.py", line 101, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/log.py", line 84, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 597, in _doReadOrWrite
    why = selectable.doRead()
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 209, in doRead
    return self._dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 215, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/basic.py", line 578, in dataReceived
    why = self.rawDataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1763, in rawDataReceived
    self._transferDecoder.dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1436, in dataReceived
    finishCallback(data[contentLength:])
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1694, in _finishRequestBody
    self.allContentReceived()
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1757, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 810, in requestReceived
    args.update(cgi.parse_multipart(self.content, pdict))
  File "/usr/lib/python3.4/cgi.py", line 235, in parse_multipart
    nextpart = b"--" + boundary
builtins.TypeError: can't concat bytes to str
<!DOCTYPE html>
<html>

<head>
   <meta charset="UTF-8">
   <script src="js/resumable.js"></script>
   <script src="js/autobahn.min.jgz"></script>
</head>

<body>
   <a href="#" id="browseButton">Select files to upload</a><br><br>
   <div id="dropFiles" style="background-color: #aaf; width: 200px; height: 200px" >Drop files here to upload</div>

   <script>
      var connection = new autobahn.Connection({
         url: "ws://localhost:8080/ws",
         realm: 'realm1'
      });
      connection.onopen = function (session, details) {
         console.log("Connected", details);

          var pinfo = null;
          function upload (args) {
              pinfo = args[0];
              console.log('upload event received', pinfo.status, pinfo.chunk, pinfo.remaining, pinfo.total, pinfo.progress);
           }
//         session.subscribe('re.upload', function (args) { //re.upload  com.example.upload.on_progress
//            var pinfo = args[0];
//            console.log('upload event received', pinfo.status, pinfo.chunk, pinfo.remaining, pinfo.total, pinfo.progress);
//         }
          session.subscribe('com.example.upload.on_progress', upload).then( //re.upload  com.example.upload.on_progress
              function (sub) {
                 console.log("subscribed to topic 'upload'" + sub);
              },
              function (err) {
                 console.log("failed to subscribe: " + err);
              }
          );

         var r = new Resumable({
            target: 'upload',
            chunkSize: 1 * 1024 * 1024,
            forceChunkSize: true, // https://github.com/23/resumable.js/issues/51
            simultaneousUploads: 4,
            testChunks: true,
            query: {
               on_progress: 're.upload',
               session: session.id
            }
         });

         if (!r.support) {
            console.log("Fatal: ResumableJS not supported!");
         } else {
            r.assignBrowse(document.getElementById('browseButton'));
            r.assignDrop(document.getElementById('dropFiles'));
            r.on('fileAdded', function (file) {
               console.log('fileAdded', file);
               r.upload();
            });
            r.on('fileSuccess', function (file, message) {
               console.log('fileSuccess', file, message);
               console.log(r.files);
               // enable repeated upload since other user can delete the file on the server
               // and this user might want to reupload the file
               file.cancel();
            });
            r.on('fileError', function (file, message) {
               console.log('fileError', file, message);
            });
         }
      };
      connection.onclose = function(reason, details) {
          console.log("Disconnected. Reason: ", reason, details.reason, details);
      };
      connection.open();
   </script>
</body>

</html>

Router Config:

{
   "controller": {
   },
   "workers": [
      {
         "id": "wamprouter",
         "type": "router",
         "options": {
            "title": "WAMP Router"
         },
         "realms": [
            {
               "name": "realm1",
               "roles": [
                  {
                     "name": "anonymous",
                     "permissions": [
                        {
                           "uri": "*",
                           "publish": true,
                           "subscribe": true,
                           "call": true,
                           "register": true
                        }
                     ]
                  },
                  {
                     "name": "backend",
                     "permissions": [
                        {
                          "uri": "mono.backend.*",
                          "call": true,
                          "register": true,
                          "subscribe": true,
                          "publish": true
                        }
                     ]
                  },
                  {
                     "name": "frontend",
                     "permissions": [
                        {
                          "uri": "mono.frontend.*",
                          "call": true,
                          "register": false,
                          "subscribe": true,
                          "publish": false
                        }
                     ]
                  }
               ]
            }
         ],
         "transports": [
            {
               "id": "web",
               "type": "web",
               "endpoint": {
                  "type": "tcp",
                  "port": 80
               },
               "paths": {
                  "/": {
                     "type": "static",
                     "directory": "../frontend"
                  },
                  "ws": {
                     "type": "websocket",
                     "url": "ws://localhost:8080/ws",

                     "debug": false,
                     "options": {
                        "compression": {
                           "deflate": {
                              "request_no_context_takeover": false,
                              "request_max_window_bits": 11,
                              "no_context_takeover": false,
                              "max_window_bits": 11,
                              "memory_level": 4
                           }
                        },
                        "auto_ping_interval": 30000,
                        "auto_ping_timeout": 30000,
                        "auto_ping_size": 4
                     }
                  },
                  "upload": {
                     "type": "upload",
                     "realm": "realm1",
                     "role": "anonymous",
                     "directory": "../frontend/uploads",
                     "temp_directory": "../frontend/temp",
                     "form_fields": {
                        "file_name": "resumableFilename",
                        "mime_type": "resumableType",
                        "total_size": "resumableTotalSize",
                        "chunk_number": "resumableChunkNumber",
                        "chunk_size": "resumableChunkSize",
                        "total_chunks": "resumableTotalChunks",
                        "content": "file",
                        "on_progress": "on_progress",
                        "session": "session"
                     },
                     "options": {
                        "max_file_size": 200000000,
                        "file_permissions": "0644",
                        "file_types": [
                           ".csv",
                           ".txt",
                           ".pdf",
                           ".img",
                           ".png",
                           ".jpg",
                           ".tar",
                           ".bz2",
                           ".xz",
                           ".gz"
                        ]
                     }
                  }
               }
            }
         ]
      }
   ]
}
goeddea commented 8 years ago

@singlecheeze - Just checked the fileupload example on my machine and it works fine.

Could you provide more details on what you're running things on, and what precisely you are doing when things break.Does the fileupload example as-is work for you on your system?

singlecheeze commented 8 years ago 
     __  __  __  __  __  __      __     __
    /  `|__)/  \/__`/__`|__) /\ |__)  |/  \
    \__,|  \\__/.__/.__/|__)/~~\|  \. |\__/

 Crossbar.io        : 0.11.2
   Autobahn         : 0.10.9
     UTF8 Validator : autobahn
     XOR Masker     : autobahn
     JSON Codec     : stdlib
     MsgPack Codec  : -
   Twisted          : 15.4.0-EPollReactor
   Python           : 3.4.0/CPython
 OS                 : Linux-3.19.0-28-generic-x86_64-with-Ubuntu-14.04-trusty
 Machine            : x86_64
singlecheeze commented 8 years ago

@goeddea Testing stock file upload and will post shortly... all I'm doing in the above is dragging and dropping a text file into the html hit box for upload.

singlecheeze commented 8 years ago

@goeddea just tested things with exact files from exmaple... same errors.

I'm using the latest resumable.js that I downloaded yesterday.

I'm thinking that maybe @hawkowl needs to look at this as from the error it looks like it might be a python 3 problem (bytes vs unicode) with twisted:

2015-10-15T10:09:15-0400 [Router      21661] Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/log.py", line 101, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/log.py", line 84, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.4/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/posixbase.py", line 597, in _doReadOrWrite
    why = selectable.doRead()
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 209, in doRead
    return self._dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/internet/tcp.py", line 215, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/basic.py", line 578, in dataReceived
    why = self.rawDataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1763, in rawDataReceived
    self._transferDecoder.dataReceived(data)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1436, in dataReceived
    finishCallback(data[contentLength:])
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1694, in _finishRequestBody
    self.allContentReceived()
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1757, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 810, in requestReceived
    args.update(cgi.parse_multipart(self.content, pdict))
  File "/usr/lib/python3.4/cgi.py", line 235, in parse_multipart
    nextpart = b"--" + boundary
builtins.TypeError: can't concat bytes to str
singlecheeze commented 8 years ago

This is what I'm seeing from the client side:

POST http://localhost:8080/upload net::ERR_EMPTY_RESPONSEResumableChunk.$.send @ resumable.js:758doneHandler @ resumable.js:697
resumable.js:758 POST http://localhost:8080/upload net::ERR_EMPTY_RESPONSEResumableChunk.$.send @ resumable.js:758doneHandler @ resumable.js:697
hawkowl commented 8 years ago

Hi, this is a Py3 compatibility bug. Thanks for reporting it. I'll get to the bottom of it tomorrow :)

singlecheeze commented 8 years ago

Win! Thank you so much!!! Using this on a big project right now and it would be awesome to get it working! +1 if you show an example with Telerik's Upload JQuery control ;) ;) ha ha ha

http://docs.telerik.com/kendo-ui/web/upload/metadata http://demos.telerik.com/kendo-ui/upload/async

singlecheeze commented 8 years ago

Hey @hawkowl any luck?

hawkowl commented 8 years ago

@singlecheeze I've found the bug, just trying to write a test to make sure it's fixed for good. Since writing it is a little harder than I thought, I'll push forward with a non-unit tested patch and follow up with tests later.

hawkowl commented 8 years ago

Ugh, looks like there's also a related bug in Twisted. I'll file an upstream ticket and patch it.

hawkowl commented 8 years ago

Okay, so there's two portions. There's a) it's broken in Twisted, as it's not ported yet, and b) it's broken in Crossbar, as there's some non-obvious Py3 incompatibilities. This functionality will only operate in Py2 for now, but a Twisted release is coming up soon, so it should be able to work very soon.

singlecheeze commented 8 years ago

Oh man I hope soon :( though I REALLY appreciate your effort and tracking of this one!

hawkowl commented 8 years ago

I've got a fix up for review in https://twistedmatrix.com/trac/ticket/8052.

hawkowl commented 8 years ago

I've also put up a Crossbar patch for review here, sans tests for now: https://github.com/crossbario/crossbar/pull/486

If you use that Twisted and that Crossbar, does it work for you, @singlecheeze?

singlecheeze commented 8 years ago

I'll check out as soon as I can. Thank you soooooo much!!! On Oct 21, 2015 2:17 AM, "Amber Brown" notifications@github.com wrote:

I've also put up a Crossbar patch for review here, sans tests for now:

486 https://github.com/crossbario/crossbar/pull/486

If you use that Twisted and that Crossbar, does it work for you, @singlecheeze https://github.com/singlecheeze?

— Reply to this email directly or view it on GitHub https://github.com/crossbario/crossbar/issues/481#issuecomment-149793395 .

singlecheeze commented 8 years ago

@hawkowl I'm not too good at github... I tried to install the latest versions of twisted and then crossbar like this but the upload still doesn't work... do you have to merge the fixes before I can run the below commands?

sudo pip3 install -U https://github.com/twisted/twisted/archive/trunk.zip
sudo pip3 install -U https://github.com/crossbario/crossbar/zipball/master
hawkowl commented 8 years ago

@singlecheeze Try this:

virtualenv -p python3 py3venv
. py3venv/bin/activate
pip install https://github.com/twisted/twisted/archive/multipart-py3-8052.zip
pip install https://github.com/hawkowl/crossbar/archive/fileupload-py3-fix.zip
crossbar start
hawkowl commented 8 years ago

This will create a virtualenv for py3, install the two fixes branches inside it, and then let you try it out in there.

hawkowl commented 8 years ago

This is merged in Crossbar and Twisted, just need to wait for releases of both. I need to write some tests for it, so it keeps working, however.

singlecheeze commented 8 years ago

Thank you! :heart_eyes:

hawkowl commented 8 years ago

(note for @oberstet / @meejah -- the base issue is fixed, but I'm not going to close this ticket until I'm sure (that is -- that it's tested ;) )

WebDevelopNemanja commented 8 years ago

Hello, I'm receiving this error, it seems that is not fixed completely... **Versions:

When I want to upload a file, I'm receiving this error for every chunk.

2016-05-22T15:41:52+0200 [Router 13911] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/twisted/web/http.py", line 1439, in dataReceived finishCallback(data[contentLength:]) File "/usr/local/lib/python3.5/dist-packages/twisted/web/http.py", line 1704, in _finishRequestBody self.allContentReceived() File "/usr/local/lib/python3.5/dist-packages/twisted/web/http.py", line 1767, in allContentReceived req.requestReceived(command, path, version) File "/usr/local/lib/python3.5/dist-packages/twisted/web/http.py", line 768, in requestReceived self.process() --- --- File "/usr/local/lib/python3.5/dist-packages/twisted/web/server.py", line 183, in process self.render(resrc) File "/usr/local/lib/python3.5/dist-packages/twisted/web/server.py", line 234, in render body = resrc.render(self) File "/usr/local/lib/python3.5/dist-packages/twisted/web/resource.py", line 250, in render return m(request) File "/opt/crossbar-repo/crossbar/twisted/fileupload.py", line 242, in render_POST fileTempDir = os.path.join(self._tempDirRoot, fileId) File "/usr/lib/python3.5/posixpath.py", line 89, in join genericpath._check_arg_types('join', a, *p) File "/usr/lib/python3.5/genericpath.py", line 145, in _check_arg_types raise TypeError("Can't mix strings and bytes in path components") from None builtins.TypeError: Can't mix strings and bytes in path components

oberstet commented 8 years ago

We will remove the file upload service. It is unfinished and broken