search

dokku / dokku-api

Unmaintained: HTTP API on top of Dokku Daemon
Apache License 2.0
58 stars 15 forks source link

JSON parse error when using dokku clone plugin via api #10

Closed thoresuenert closed 6 years ago

thoresuenert commented 6 years ago

When we send a clone command via API the response from unixsocket includes ansi/ascii escape sequences (ex. \x1b[1G)

When logged the results shows as:

       \n3T17:53:29.468641187Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.468644801Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Resolving node version 4.0.0...\nr.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Resolving node version 4.0.0...\nr.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Downloading and installing node 4.0.0...\nID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Downloading and installing node 4.0.0...\nID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Using default npm version: 2.14.2\n1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Using default npm version: 2.14.2\n1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.468670167Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.468674901Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Restoring cache\n78712Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Restoring cache\n01772Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Loading 2 from cacheDirectories (default):\n-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Loading 2 from cacheDirectories (default):\n-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       - node_modules\n060073Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       - node_modules\n073526Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       - bower_components (not cached - skipping)\n-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       - bower_components (not cached - skipping)\n-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.469084666Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.469088184Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Building dependencies\n app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Building dependencies\n app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Installing node modules (package.json)\n TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       Installing node modules (package.json)\n TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.469128261Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
       \n3T17:53:29.469131743Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Caching build\n9135233Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:
-----> Caching build\n9138821Z app[worker.1]: 8 TID-gr4rqfni4 CommandRunner JID-1c54ccf2e6cb0e6ceaa5d86a INFO: [CommandRunner] line:

we see it in postgres as:

 {"ok":true,"output":"Creating uu... done\n-----> Creating uu from https://github.com/heroku/node-js-sample.git master\n\x1B[1G-----> Cleaning up...\n\x1B[1G-----> Building uu from hero
kuish...\n\x1B[1G-----> Adding BUILD_ENV to build environment...\n\x1B[1G\x1B[1G       \x1B[1G-----> Node.js app detected\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Creating runtime e
nvironment\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G       NPM_CONFIG_LOGLEVEL=error\n\x1B[1G\x1B[1G       NPM_CONFIG_PRODUCTION=true\n\x1B[1G\x1B[1G       NODE_VERBOSE=false\n\x1B[1G\x1B[
1G       NODE_ENV=production\n\x1B[1G\x1B[1G       NODE_MODULES_CACHE=true\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Installing binaries\n\x1B[1G\x1B[1G       engines.node (package.j
son):  4.0.0\n\x1B[1G\x1B[1G       engines.npm (package.json):   unspecified (use default)\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G       Resolving node version 4.0.0...\n\x1B[1G\x1B[1G
     Downloading and installing node 4.0.0...\n\x1B[1G\x1B[1G       Using default npm version: 2.14.2\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Restoring cache\n\x1B[1G\x1B[1G
Skipping cache restore (not-found)\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Building dependencies\n\x1B[1G\x1B[1G       Installing node modules (package.json)\n\x1B[1G\x1B[1G
express@4.16.2 node_modules/express\n\x1B[1G\x1B[1G       ├── escape-html@1.0.3\n\x1B[1G\x1B[1G       ├── array-flatten@1.1.1\n\x1B[1G\x1B[1G       ├── setprototypeof@1.1.0\n\x1B[1G\x1B
[1G       ├── cookie-signature@1.0.6\n\x1B[1G\x1B[1G       ├── utils-merge@1.0.1\n\x1B[1G\x1B[1G       ├── merge-descriptors@1.0.1\n\x1B[1G\x1B[1G       ├── methods@1.1.2\n\x1B[1G\x1B[1
G       ├── path-to-regexp@0.1.7\n\x1B[1G\x1B[1G       ├── encodeurl@1.0.1\n\x1B[1G\x1B[1G       ├── range-parser@1.2.0\n\x1B[1G\x1B[1G       ├── vary@1.1.2\n\x1B[1G\x1B[1G       ├── fr
esh@0.5.2\n\x1B[1G\x1B[1G       ├── parseurl@1.3.2\n\x1B[1G\x1B[1G       ├── content-type@1.0.4\n\x1B[1G\x1B[1G       ├── etag@1.8.1\n\x1B[1G\x1B[1G       ├── statuses@1.3.1\n\x1B[1G\x1
B[1G       ├── cookie@0.3.1\n\x1B[1G\x1B[1G       ├── content-disposition@0.5.2\n\x1B[1G\x1B[1G       ├── serve-static@1.13.1\n\x1B[1G\x1B[1G       ├── safe-buffer@5.1.1\n\x1B[1G\x1B[1G
       ├── depd@1.1.2\n\x1B[1G\x1B[1G       ├── on-finished@2.3.0 (ee-first@1.1.1)\n\x1B[1G\x1B[1G       ├── finalhandler@1.1.0 (unpipe@1.0.0)\n\x1B[1G\x1B[1G       ├── debug@2.6.9 (ms@
2.0.0)\n\x1B[1G\x1B[1G       ├── qs@6.5.1\n\x1B[1G\x1B[1G       ├── proxy-addr@2.0.2 (forwarded@0.1.2, ipaddr.js@1.5.2)\n\x1B[1G\x1B[1G       ├── send@0.16.1 (destroy@1.0.4, ms@2.0.0, m
ime@1.4.1, http-errors@1.6.2)\n\x1B[1G\x1B[1G       ├── type-is@1.6.15 (media-typer@0.3.0, mime-types@2.1.17)\n\x1B[1G\x1B[1G       ├── accepts@1.3.4 (negotiator@0.6.1, mime-types@2.1.1
7)\n\x1B[1G\x1B[1G       └── body-parser@1.18.2 (bytes@3.0.0, raw-body@2.3.2, http-errors@1.6.2, iconv-lite@0.4.19)\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Caching build\n\x1B[1G\x
1B[1G       Clearing previous node cache\n\x1B[1G\x1B[1G       Saving 2 cacheDirectories (default):\n\x1B[1G\x1B[1G       - node_modules\n\x1B[1G\x1B[1G       - bower_components (nothin
g to cache)\n\x1B[1G\x1B[1G       \n\x1B[1G\x1B[1G-----> Build succeeded!\n\x1B[1G\x1B[1G       \x1B[1G-----> Discovering process types\n\x1B[1G\x1B[1G       Procfile declares types ->
web\n\x1B[1G-----> Releasing uu (dokku/uu:latest)...\n\x1B[1G-----> Deploying uu (dokku/uu:latest)...\n\x1B[1G-----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
\n\x1B[1G-----> App Procfile file found (/home/dokku/uu/DOKKU_PROCFILE)\n\x1B[1G-----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...\n\x1B[1G-----> New DO
KKU_SCALE file generated\n\x1B[1G=====> web=1\n\x1B[1G-----> Attempting pre-flight checks\n\x1B[1G       For more efficient zero downtime deployments, create a file CHECKS.\n\x1B[1G
   See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples\n\x1B[1G       CHECKS file not found in container: Running simple container check...\n\x1B[1G----->
Waiting for 10 seconds ...\n\x1B[1G-----> Default container check successful!\n\x1B[1G-----> Running post-deploy\n\x1B[1G=====> renaming container (19108dd968c1) naughty_agnesi to uu.we
b.1\n\x1B[1G-----> Creating new /home/dokku/uu/VHOST...\n\x1B[1G-----> Setting config vars\n\x1B[1G       DOKKU_NGINX_PORT:  80\n\x1B[1G-----> Setting config vars\n\x1B[1G       DOKKU_P
ROXY_PORT_MAP:  http:80:5000\n\x1B[1G-----> Configuring uu.louvi.de...(using built-in template)\n\x1B[1G-----> Creating http nginx.conf\n\x1B[1G-----> Running nginx-pre-reload\n\x1B[1G
      Reloading nginx\n\x1B[1G-----> Setting config vars\n\x1B[1G       DOKKU_APP_RESTORE:  1\n\x1B[1G-----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)\n\x1B[
1G=====> Application deployed:\n\x1B[1G "}

i tried everything i can over the last two days to remove those chars.

i tried different ways of encode, scrub, gsub etc. but i cannot fix this problem.

PS: i found socket.gets("\n") gives the whole result at once, and sockets.gets('\n') gives the result line by line.

I don't know enough about ruby, string interpolation etc. to solve with problem. Any ideas?

beydogan commented 6 years ago

@thoresuenert thank you for report, I'm able to reproduce the issue, will fix it shortly.

thoresuenert commented 6 years ago

@beydogan let me know if i can help

beydogan commented 6 years ago

This is due to unicode chars in result data from dokku-daemon. Git inside dokku clone prints string containing \e[1G. And its breaking JSON.parse(...). I've spent more time than I should've on this and couldn't find a good way to catch all cases but to fix this one for now, I just removed \e[1G from strings, so it should be fine. Let me know if you have any issues on this.

thoresuenert commented 6 years ago

@beydoga When i use your daemon implementation in go, could that avoid this problem? Savant gave me a hint because bash and not bash

beydogan commented 6 years ago

@thoresuenert no its the same with go daemon, im using that one.

Didnt last commit fix issue?

On Jan 30, 2018 6:11 PM, "thoresuenert" notifications@github.com wrote:

@beydoga When i use your daemon implementation in go, could that avoid this problem? Savant gave me a hint because bash and not bash

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/dokku/dokku-api/issues/10#issuecomment-361623069, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr4CgOTtQljJFS8hponsdcElhGtRuUcks5tPzEGgaJpZM4RdndQ .

thoresuenert commented 6 years ago

Not tried yet. Thanks for fixing it. Will test tomorrow.