search

kalisio / k-vigicrues

A Krawler based service to download data from French flood warning system Vigicrues
MIT License
3 stars 0 forks source link

job stations fail #9

Closed cnouguier closed 5 years ago

cnouguier commented 5 years ago

When deployed on the server and cleaning the database and the output directories, the job fails with the following trace:

  krawler:cli Initializing krawler application +0ms
  krawler:cli API prefix  +25ms
  krawler:hooks Adding hook unzipFromStore to hook chain +0ms
  krawler:hooks Adding hook readJson to hook chain +0ms
  krawler:hooks Adding hook writeJson to hook chain +0ms
  krawler:hooks Adding hook clearData to hook chain +0ms
  krawler:hooks Adding hook createStores to hook chain +1ms
  krawler:hooks Adding hook removeStores to hook chain +0ms
  krawler:hooks Adding hook removeStores to hook chain +1ms
Launching job vigicrues-stations at 2018-12-17T09:53:44.732Z, please wait...
  krawler:hooks Executing hook createStores not filtered by { skip: { '$exists': false } } +23ms
  krawler:hooks:store Looking for store memory +0ms
  krawler:stores Can't find store with ID memory +0ms
  krawler:hooks:store Creating store for vigicrues-stations with options  { id: 'memory' } +1ms
  krawler:stores Creating memory store memory with following parameters undefined +0ms
  krawler:hooks:store Looking for store fs +2ms
  krawler:stores Can't find store with ID fs +2ms
  krawler:hooks:store Creating store for vigicrues-stations with options  { id: 'fs', options: { path: '/home/ubuntu/krawler/sandbox' } } +0ms
  krawler:stores Creating FS store fs with following parameters { path: '/home/ubuntu/krawler/sandbox' } +0ms
  krawler:utils Seeking for store in service, data or params with path/name store +0ms
  krawler:jobs Launching job with following template {} +0ms
  krawler:jobs Creating async job vigicrues-stations with following options { workersLimit: 1, faultTolerant: false } +0ms
  krawler:tasks Creating task vigicrues/stations +0ms
  krawler:tasks Requesting http://services.sandre.eaufrance.fr/geo/hyd with following parameters { version: '1.1.0',
  typename: 'StationHydro_FXX',
  outputFormat: 'geojson',
  service: 'WFS',
  request: 'GetFeature' } +0ms
  krawler:utils Seeking for store in service, data or params with path/name store +8ms
  krawler:hooks Executing hook unzipFromStore not filtered by { skip: { '$exists': false } } +2s
  krawler:utils Seeking for store in hook unzipFromStore +2s
  krawler:utils Seeking for store in service, data or params with path/name fs +1ms
  krawler:hooks:store Unzipping from store { key: 'vigicrues/stations', store: 'fs' } { path: 'vigicrues', store: 'fs' } +2s
  krawler:hooks Executing hook readJson not filtered by { skip: { '$exists': false } } +7ms
  krawler:utils Seeking for store in hook readJson +5ms
  krawler:utils Seeking for store in service, data or params with path/name store +0ms
  krawler:hooks:json Reading JSON file /home/ubuntu/krawler/sandbox/vigicrues/StationHydro_FXX-geojson.dat +0ms
  krawler:jobs Some tasks failed { Error: ENOENT: no such file or directory, open '/home/ubuntu/krawler/sandbox/vigicrues/StationHydro_FXX-geojson.dat'
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/ubuntu/krawler/sandbox/vigicrues/StationHydro_FXX-geojson.dat',
  hook:
   { type: 'after',
     arguments: [ [Object], [Object] ],
     service:
      { events: [Array],
        generators: [Object],
        storesService: [Object],
        create: [Function: newMethod],
        remove: [Function: newMethod],
        methods: [Object],
        hooks: [Function: hooks],
        domain: undefined,
        _events: [Object],
        _maxListeners: undefined,
        setMaxListeners: [Function: setMaxListeners],
        getMaxListeners: [Function: getMaxListeners],
        emit: [Function: emit],
        addListener: [Function: addListener],
        on: [Function: addListener],
        prependListener: [Function: prependListener],
        once: [Function: once],
        prependOnceListener: [Function: prependOnceListener],
        removeListener: [Function: removeListener],
        removeAllListeners: [Function: removeAllListeners],
        listeners: [Function: listeners],
        listenerCount: [Function: listenerCount],
        eventNames: [Function: eventNames],
        publish: [Function: publish],
        _eventsCount: 3,
        _super: undefined,
        [Symbol(@feathersjs/transport-commons/publishers)]: {} },
     app:
      { [EventEmitter: app]
        domain: undefined,
        _events: [Object],
        _maxListeners: undefined,
        setMaxListeners: [Function: setMaxListeners],
        getMaxListeners: [Function: getMaxListeners],
        emit: [Function: emit],
        addListener: [Function: addListener],
        on: [Function: addListener],
        prependListener: [Function: prependListener],
        once: [Function: once],
        prependOnceListener: [Function: prependOnceListener],
        removeListener: [Function: removeListener],
        removeAllListeners: [Function: removeAllListeners],
        listeners: [Function: listeners],
        listenerCount: [Function: listenerCount],
        eventNames: [Function: eventNames],
        init: [Function: init],
        defaultConfiguration: [Function: defaultConfiguration],
        lazyrouter: [Function: lazyrouter],
        handle: [Function: handle],
        use: [Function: newMethod],
        route: [Function: route],
        engine: [Function: engine],
        param: [Function: param],
        set: [Function: set],
        path: [Function: path],
        enabled: [Function: enabled],
        disabled: [Function: disabled],
        enable: [Function: enable],
        disable: [Function: disable],
        acl: [Function],
        bind: [Function],
        checkout: [Function],
        connect: [Function],
        copy: [Function],
        delete: [Function],
        get: [Function],
        head: [Function],
        link: [Function],
        lock: [Function],
        'm-search': [Function],
        merge: [Function],
        mkactivity: [Function],
        mkcalendar: [Function],
        mkcol: [Function],
        move: [Function],
        notify: [Function],
        options: [Function],
        patch: [Function],
        post: [Function],
        propfind: [Function],
        proppatch: [Function],
        purge: [Function],
        put: [Function],
        rebind: [Function],
        report: [Function],
        search: [Function],
        source: [Function],
        subscribe: [Function],
        trace: [Function],
        unbind: [Function],
        unlink: [Function],
        unlock: [Function],
        unsubscribe: [Function],
        all: [Function: all],
        del: [Function],
        render: [Function: render],
        listen: [Function: newMethod],
        request: [Object],
        response: [Object],
        cache: {},
        engines: {},
        settings: [Object],
        _eventsCount: 2,
        locals: [Object],
        mountpath: '/',
        configure: [Function: configure],
        service: [Function: service],
        setup: [Function: newMethod],
        version: '3.2.1',
        methods: [Array],
        mixins: [Array],
        services: [Object],
        providers: [Array],
        _setup: false,
        hookTypes: [Array],
        hooks: [Function: hooks],
        eventMappings: [Object],
        _super: undefined,
        _router: [Object],
        rest: [Object],
        channel: [Function: channel],
        publish: [Function: publish],
        lookup: [Function: lookup],
        io: [Object],
        _isSetup: true,
        [Symbol(@feathersjs/transport-commons/channels)]: {},
        [Symbol(@feathersjs/transport-commons/publishers)]: {},
        [Symbol(@feathersjs/transport-commons/router)]: [Object] },
     method: 'create',
     path: 'tasks',
     params: { store: [Object] },
     data:
      { options: [Object],
        id: 'vigicrues/stations',
        type: 'wfs',
        intermediate: [Array] },
     result:
      { options: [Object],
        id: 'vigicrues/stations',
        type: 'wfs',
        intermediate: [Array] } } } +2s
Cannot read property 'hasOwnProperty' of undefined

It seems we ara facing an asynchronous issue as the hook readJson is trying to read a file which is still not available

claustres commented 5 years ago

The final error is related to https://github.com/kalisio/krawler/issues/47#issuecomment-447763959 because the job fails. But the root cause of the failure is probably that the request to the SANDRE service has not generated a good output or has not been unzipped properly.

claustres commented 5 years ago

Related and should be closed by https://github.com/kalisio/krawler/issues/52.

cnouguier commented 5 years ago

The job is now running