search

Kaixhin / FGLab

Future Gadget Laboratory
https://kaixhin.github.io/FGLab/
MIT License
223 stars 34 forks source link

Heroku App/Docker: cannot use any FGMachines ({"error":"No machine capacity available"}) #20

Closed asmith26 closed 7 years ago

asmith26 commented 8 years ago

I've read all documentation and was trying to run the Bayesian Optimisation example through the Heroku app, but at Step 5 of the "Quickstart" of the github.io documentation, when I click the add button/drop-down menu nothing happens (no options, messages, errors...). Indeed, when I try and Submit a new experiment I get {"error":"No machine capacity available"}

Thus I tried to deploy FGLab and FGMachine locally via docker, but I'm still having problems with FGMachines:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
acf5d5ea7a73        kaixhin/fgmachine   "node machine"           10 minutes ago      Up 10 minutes       0.0.0.0:5081->5081/tcp   fgmachine
7523ba457713        kaixhin/fglab       "node lab"               2 hours ago         Up 2 hours          0.0.0.0:5080->5080/tcp   fglab
4e01d03b0ae2        mongo               "/entrypoint.sh mongo"   2 hours ago         Up 2 hours          27017/tcp                mongodb

$ docker logs fgmachine
Server listening on port null
{ RequestError: Error: Invalid protocol: <MY LAN IP>:
    at new RequestError (/root/FGMachine/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/root/FGMachine/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/root/FGMachine/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/root/FGMachine/node_modules/request/request.js:187:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.init (/root/FGMachine/node_modules/request/request.js:460:17)
    at Request.RP$initInterceptor [as init] (/root/FGMachine/node_modules/request-promise-core/configure/request2.js:45:29)
    at new Request (/root/FGMachine/node_modules/request/request.js:129:8)
    at request (/root/FGMachine/node_modules/request/index.js:55:10)
    at fs.readFile.then.catch (/root/FGMachine/machine.js:80:3)
  name: 'RequestError',
  message: 'Error: Invalid protocol: <MY LAN IP>:',
  cause: 
   Error: Invalid protocol: <MY LAN IP>:
       at Request.init (/root/FGMachine/node_modules/request/request.js:460:31)
       at Request.RP$initInterceptor [as init] (/root/FGMachine/node_modules/request-promise-core/configure/request2.js:45:29)
       at new Request (/root/FGMachine/node_modules/request/request.js:129:8)
       at request (/root/FGMachine/node_modules/request/index.js:55:10)
       at fs.readFile.then.catch (/root/FGMachine/machine.js:80:3),
  error: 
   Error: Invalid protocol: <MY LAN IP>:
       at Request.init (/root/FGMachine/node_modules/request/request.js:460:31)
       at Request.RP$initInterceptor [as init] (/root/FGMachine/node_modules/request-promise-core/configure/request2.js:45:29)
       at new Request (/root/FGMachine/node_modules/request/request.js:129:8)
       at request (/root/FGMachine/node_modules/request/index.js:55:10)
       at fs.readFile.then.catch (/root/FGMachine/machine.js:80:3),
  options: 
   { uri: '<MY LAN IP>:5080/api/v1/machines',
     method: 'POST',
     json: 
      { address: 'localhost:5081',
        hostname: 'asmith26',
        os: [Object],
        cpus: [Object],
        mem: '7.72GB',
        gpus: [] },
     gzip: true,
     callback: [Function: RP$callback],
     transform: undefined,
     simple: true,
     resolveWithFullResponse: false,
     transform2xxOnly: false },
  response: undefined }

I've tried:

 docker run -d --name fgmachine -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -e FGLAB_URL=<MY LAN IP>:5080 -e FGMACHINE_URL=<MY LAN IP>:5081 -p 5081:5081 kaixhin/fgmachine

and (difference is FGMACHINE_URL=localhost:5081)

docker run -d --name fgmachine -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -e FGLAB_URL=<MY LAN IP>:5080 -e FGMACHINE_URL=localhost:5081 -p 5081:5081 kaixhin/fgmachine

Thank you for any help in advance!

Kaixhin commented 8 years ago

You've missed out the http:// in the URLs. Additionally, because of the networking setup with Docker it's recommended to use an accessible IP address/hostname rather than localhost.

asmith26 commented 8 years ago

Thank you for your help - running (with http://) docker run -d --name fgmachine -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -e FGLAB_URL=http://<MY LAN IP>:5080 -e FGMACHINE_URL=http://<MY LAN IP>:5081 -p 5081:5081 kaixhin/fgmachine that has fixed my Invalid protocol error :-)

Unfortunately I still can't add this FGMachine to run the Bayesian Optimisation experiment via the FGLab GUI (i.e. clicking the add button/drop-down menu nothing happens (no options, messages, errors...)).

Running docker logs fgmachine initially, I get just Server listening on port 5081. However after a few minutes my log then gets appended with:

{ RequestError: Error: connect ETIMEDOUT <MY LAN IP>:5080
    at new RequestError (/root/FGMachine/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/root/FGMachine/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/root/FGMachine/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/root/FGMachine/node_modules/request/request.js:187:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.onRequestError (/root/FGMachine/node_modules/request/request.js:813:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:308:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1272:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
  name: 'RequestError',
  message: 'Error: connect ETIMEDOUT <MY LAN IP>:5080',
  cause: 
   { Error: connect ETIMEDOUT <MY LAN IP>:5080
       at Object.exports._errnoException (util.js:1026:11)
       at exports._exceptionWithHostPort (util.js:1049:20)
       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)
     code: 'ETIMEDOUT',
     errno: 'ETIMEDOUT',
     syscall: 'connect',
     address: '<MY LAN IP>',
     port: 5080 },
  error: 
   { Error: connect ETIMEDOUT <MY LAN IP>:5080
       at Object.exports._errnoException (util.js:1026:11)
       at exports._exceptionWithHostPort (util.js:1049:20)
       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)
     code: 'ETIMEDOUT',
     errno: 'ETIMEDOUT',
     syscall: 'connect',
     address: '<MY LAN IP>',
     port: 5080 },
  options: 
   { uri: 'http://<MY LAN IP>:5080/api/v1/machines',
     method: 'POST',
     json: 
      { address: 'http://<MY LAN IP>:5081',
        hostname: 'asmith26',
        os: [Object],
        cpus: [Object],
        mem: '7.72GB',
        gpus: [] },
     gzip: true,
     callback: [Function: RP$callback],
     transform: undefined,
     simple: true,
     resolveWithFullResponse: false,
     transform2xxOnly: false },
  response: undefined }

As far as I can tell, this appears to be some timeout error regarding the linking of a docker fgmachine to a docker fglab (??). I'm deploying all docker containers on my one desktop computer (if it helps). Also, http://localhost:5080/api/v1/machines yields just [].

Thanks again for any help!

Kaixhin commented 8 years ago

With docker logs fgmachine you should be seeing:

Server listening on port 5081
Registered with FGLab successfully
Projects registered with FGLab successfully

Corresponding to the following with docker logs fglab:

<MAC ADDR> - - [07/Sep/2016:16:41:39 +0000] "POST /api/v1/machines HTTP/1.1" 201 623
<MAC ADDR> - - [07/Sep/2016:16:41:39 +0000] "POST /api/v1/machines/57d043439ba0300001c3d76b/projects HTTP/1.1" 200 15

Something is going wrong when trying to register the machine, so there is probably something unusual with your network setup. For instance, if you have proxies you can try experimenting with these environment variables.

asmith26 commented 7 years ago

It turned out to be my Firewall - adding an exception rule fixed the problem. Thanks again for your help.