client crashes if server is down

boxxxie commented 9 years ago

i used your examples, only started 1 server, put the client in an interval loop. kept getting errors for the server that wasn't up. i see a blacklist in your code, but maybe it's not used for this purpose. if it is then it's failing.

after a minute or so my client crashes.

maybe the queue should kick clients out that error.

Trello Card

boxxxie commented 9 years ago

2015-03-20T22:26:43.531Z        l1s5n4kc585z/1426890365697/10443/-      ERROR   act     queue   -       OUT     nc1vfm7d6gd5/nmfpj145jiow       role:queue,cmd:enqueue  33383   {role=queue,msg={task=my task,param=3},hook=enqueue,queue=q2,type=remote}      ENTRY   A;62wr9q2oo7n5  -       seneca: Action role:queue,cmd:enqueue failed: [TIMEOUT].        act_execute     {id=nc1vfm7d6gd5/nmfpj145jiow,gate=false,ungate=false,desc=role:queue,cmd:enqueue,plugin={name=queue,tag=-},time={start=1426890370152,end=1426890403531},message=[TIMEOUT],pattern=role:queue,cmd:enqueue,instance=Seneca/0.6.1/l1s5n4kc585z/1426890365697/10443/-}   Error: [TIMEOUT]
    at null._onTimeout (/home/paul/onist/seneca-redist-queue-testing/node_modules/seneca/node_modules/gate-executor/gate-executor.js:95:21)
    at Timer.listOnTimeout (timers.js:110:15)
2015-03-20T22:26:43.532Z        l1s5n4kc585z/1426890365697/10443/-      ERROR   plugin  transport               ACT     0786ovoeyccm/0kxkxe8vd4nw       role:transport,hook:client,type:web   client   no-message-id   {type=web,port=8082,host=0.0.0.0,path=/act,protocol=http,timeout=500,pin=cmd:enqueue-remote,queue:q2,msgprefix=seneca_,callmax=111111,msgidlen=12,role=transport,hook=client}  {kind=res,res=,error={code=ECONNREFUSED,errno=ECONNREFUSED,syscall=connect},time={client_recv=1426890403532}}
2015-03-20T22:26:43.533Z        l1s5n4kc585z/1426890365697/10443/-      ERROR   plugin  transport               ACT     0786ovoeyccm/0kxkxe8vd4nw       role:transport,hook:client,type:web   client   no-message-id   {type=web,port=8082,host=0.0.0.0,path=/act,protocol=http,timeout=500,pin=cmd:enqueue-remote,queue:q2,msgprefix=seneca_,callmax=111111,msgidlen=12,role=transport,hook=client}  {kind=res,res=,error={code=ECONNREFUSED,errno=ECONNREFUSED,syscall=connect},time={client_recv=1426890403533}}

Seneca Fatal Error
==================

Message: seneca: Action role:queue,hook:enqueue,type:remote,queue:q2 failed: [TIMEOUT].

Code: act_execute

Details: { id: 'o2sbb5psk955/taqhnre207mc',
  gate: false,
  ungate: true,
  desc: 'role:queue,hook:enqueue,type:remote,queue:q2',
  cb: [Function],
  plugin: { name: 'queue', tag: '-' },
  fn: [Function],
  time: { start: 1426890370058, end: 1426890403392 },
  'orig$': { [Error: [TIMEOUT]] timeout: true },
  'message$': '[TIMEOUT]',
  message: '[TIMEOUT]',
  pattern: 'role:queue,hook:enqueue,type:remote,queue:q2',
  instance: 'Seneca/0.6.1/l1s5n4kc585z/1426890365697/10443/-' }

Stack:
    at null._onTimeout (/home/paul/onist/seneca-redist-queue-testing/node_modules/seneca/node_modules/gate-executor/gate-executor.js:95:21)
    at Timer.listOnTimeout (timers.js:110:15)

Instance: Seneca/0.6.1/l1s5n4kc585z/1426890365697/10443/-
  ALL ERRORS FATAL: action called with argument fatal$:true (probably a plugin init error, or using a plugin seneca instance, see senecajs.org/fatal.html)
    at act_done (/home/paul/onist/seneca-redist-queue-testing/node_modules/seneca/seneca.js:1423:29)

When: 2015-03-20T22:26:43.395Z

Log: [sys,seneca,0.6.1,l1s5n4kc585z/1426890365697/10443/-,act_execute,seneca: Action role:queue,hook:enqueue,type:remote,queue:q2 failed: [TIMEOUT].,{id=o2sbb5psk955/taqhnre207mc,gate=false,ungate=true,desc=role:queue,hook:enqueue,type:remote,queue:q2,plugin={name=queue,tag=-},time={start=1426890370058,end=1426890403392},message=[TIMEOUT],pattern=role:queue,hook:enqueue,type:remote,queue:q2,instance=Seneca/0.6.1/l1s5n4kc585z/1426890365697/10443/-},all-errors-fatal]

Node:
  { http_parser: '2.3', node: '0.12.0', v8: '3.28.73', uv: '1.0.2', zlib: '1.2.8', modules: '14', openssl: '1.0.2' },
  { debug: false, uv: true, ipv6: true, tls_npn: true, tls_sni: true, tls_ocsp: true, tls: true },
  [ 'Binding contextify', 'Binding natives', 'NativeModule events', 'NativeModule util', 'NativeModule buffer', 'Binding buffer', 'Binding smalloc', 'NativeModule path', 'NativeModule module', 'NativeModule vm', 'NativeModule assert', 'NativeModule fs', 'Binding fs', 'Binding constants', 'NativeModule stream', 'NativeModule _stream_readable', 'NativeModule _stream_writable', 'NativeModule _stream_duplex', 'NativeModule _stream_transform', 'NativeModule _stream_passthrough', 'NativeModule smalloc', 'NativeModule net', 'NativeModule timers', 'Binding timer_wrap', 'NativeModule _linklist', 'Binding cares_wrap', 'Binding uv', 'Binding pipe_wrap', 'Binding tcp_wrap', 'Binding stream_wrap', 'NativeModule readline', 'NativeModule console', 'Binding tty_wrap', 'NativeModule tty', 'Binding signal_wrap', 'NativeModule domain', 'NativeModule http', 'NativeModule _http_incoming', 'NativeModule _http_common', 'NativeModule freelist', 'Binding http_parser', 'NativeModule _http_outgoing', 'NativeModule _http_server', 'NativeModule _http_agent', 'NativeModule _http_client', 'NativeModule url', 'NativeModule punycode', 'NativeModule querystring', 'NativeModule https', 'NativeModule tls', 'NativeModule _tls_common', 'NativeModule constants', 'Binding crypto', 'NativeModule _tls_wrap', 'NativeModule crypto', 'NativeModule string_decoder', 'Binding tls_wrap', 'NativeModule _tls_legacy', 'NativeModule zlib', 'Binding zlib', 'NativeModule dns' ]

Process:
  pid=10443, arch=x64, platform=linux,
  path=/usr/bin/node,
  argv=[ 'node',  '/home/paul/onist/seneca-redist-queue-testing/client1.js' ],
  env={ CMD_DURATION: '17.8s',  DBUS_SESSION_BUS_ADDRESS: 'unix:abstract=/tmp/dbus-nDLMdxF3dN,guid=d4cff602a353748d65fbddd054f643f4',  DESKTOP_SESSION: 'gnome',  DISPLAY: ':0',  GDMSESSION: 'gnome',  GDM_LANG: 'en_US.utf8',  GJS_DEBUG_OUTPUT: 'stderr',  GJS_DEBUG_TOPICS: 'JS ERROR;JS LOG',  GNOME_DESKTOP_SESSION_ID: 'this-is-deprecated',  GPG_AGENT_INFO: '/run/user/1000/keyring/gpg:0:1',  HOME: '/home/paul',  LANG: 'en_US.UTF-8',  LC_MEASUREMENT: 'en_US.utf8',  LC_MONETARY: 'en_US.utf8',  LC_NUMERIC: 'en_US.utf8',  LC_PAPER: 'en_US.utf8',  LC_TIME: 'en_US.utf8',  LOGNAME: 'paul',  LS_COLORS: 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:',  MAIL: '/var/spool/mail/paul',  MOZ_PLUGIN_PATH: '/usr/lib/mozilla/plugins',  PATH: '/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/bin:/sbin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/paul/.gem/ruby/2.2.0/bin',  PWD: '/home/paul/onist/seneca-redist-queue-testing',  SESSION_MANAGER: 'local/thebrick:@/tmp/.ICE-unix/536,unix/thebrick:/tmp/.ICE-unix/536',  SHELL: '/usr/bin/fish',  SHLVL: '2',  SSH_AGENT_PID: '552',  SSH_ASKPASS: '/usr/lib/seahorse/seahorse-ssh-askpass',  SSH_AUTH_SOCK: '/run/user/1000/keyring/ssh',  TERM: 'screen-256color',  TMUX: '/tmp/tmux-1000/default,4343,2',  TMUX_PANE: '%6',  USER: 'paul',  USERNAME: 'paul',  VTE_VERSION: '3803',  WINDOWID: '25165830',  WINDOWPATH: '1',  XAUTHORITY: '/var/run/gdm/auth-for-paul-Oekmdk/database',  XDG_CURRENT_DESKTOP: 'GNOME',  XDG_MENU_PREFIX: 'gnome-',  XDG_RUNTIME_DIR: '/run/user/1000',  XDG_SEAT: 'seat0',  XDG_SESSION_DESKTOP: 'gnome',  XDG_SESSION_ID: 'c2',  XDG_VTNR: '1',  __fish_bin_dir: '/usr/bin',  __fish_datadir: '/usr/share/fish',  __fish_help_dir: '/usr/share/doc/fish',  __fish_runtime_dir: '/run/user/1000',  __fish_sysconfdir: '/etc/fish' }

SENECA TERMINATED at 2015-03-20T22:26:43.537Z. See above for error report.

mcollina commented 9 years ago

I should definitely point out that the transport-based driver is for development/testing and not production.

As an example, check out https://github.com/Nss/seneca-sqs-queue.

I should definitely add a link and a motivation on this issue.

paolochiodi commented 8 years ago

I think the problem here is that all action started from a plugin are considered "fatal", thus an error will cause the instance to die.

If I understand correctly this is going to change in future seneca versions.

dsaikiran commented 8 years ago

Hi, I need to catch that exception and handle it, how to capture that timeout / connection refused exceptions?

geek commented 8 years ago

@dsaikiran we should expose that as an event on seneca as timeout... something we need to add

senecajs-labs / seneca-queue

client crashes if server is down #2