search

cozy / ACH

Import/Export data from your Cozy
4 stars 5 forks source link

Big bills import crashed a docker couchdb #9

Open LucsT opened 6 years ago

LucsT commented 6 years ago

I'm using a local stack using a local couch in a docker.

When importing bills (and banks operations) from a json to this local cozy, CPUs begin to be fully used. The process start well, and seems to be limited as i have a correct output : 

io.cozy.bank.operations: 4.40%  
io.cozy.bank.operations: 8.80%  
io.cozy.bills: 0.98%  
io.cozy.bank.operations: 13.20%
io.cozy.bills: 1.46%
io.cozy.bank.operations: 17.61%
io.cozy.bills: 1.95%
io.cozy.bank.operations: 22.01%
io.cozy.bills: 2.44%
io.cozy.bank.operations: 26.41%
io.cozy.bills: 2.93%
io.cozy.bank.operations: 30.81%
io.cozy.bank.operations: 35.21%
io.cozy.bills: 3.42%
io.cozy.bills: 3.90%

Then the process crashed with some error as 500 then 503 on the couch.

Oops! An error occured.
{ FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef48003b',
     status: 500,
     statusText: 'Internal Server Error',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 7b 22 73 74 61 74 75 73 5f 63 6f 64 65 22 3a 35 30 33 2c 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 72 65 ... > ],
     _abort: false,
     _bytes: 103 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef48003b',
  status: 500,
  reason: 
   { error: 
      { status_code: 503,
        error: 'no_couch',
        reason: 'could not create connection with the server' } } }
Unhandled promise rejection.
FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
Oops! An error occured.
{ FetchError: {"error":{"status_code":503,"error":"no_couch","reason":"could not create connection with the server"}}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef456c20',
     status: 500,
     statusText: 'Internal Server Error',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 7b 22 73 74 61 74 75 73 5f 63 6f 64 65 22 3a 35 30 33 2c 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 72 65 ... > ],
     _abort: false,
     _bytes: 103 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/8814b2f08b0a24bf9479010aef456c20',
  status: 500,
  reason: 
   { error: 
      { status_code: 503,
        error: 'no_couch',
        reason: 'could not create connection with the server' } } }
Oops! An error occured.
{ FetchError: {"error":"no_couch","ok":false,"original":"Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/875a2215ddaed5d97a4c4a68c85756dd: read tcp 127.0.0.1:34157->127.0.0.1:5984: read: connection reset by peer","reason":"could not create connection with the server","status":"503"}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/875a2215ddaed5d97a4c4a68c85756dd',
     status: 503,
     statusText: 'Service Unavailable',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 6f 6b 22 3a 66 61 6c 73 65 2c 22 6f 72 69 67 69 6e 61 6c 22 3a 22 50 75 74 20 68 74 74 ... > ],
     _abort: false,
     _bytes: 305 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/875a2215ddaed5d97a4c4a68c85756dd',
  status: 503,
  reason: 
   { error: 'no_couch',
     ok: false,
     original: 'Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/875a2215ddaed5d97a4c4a68c85756dd: read tcp 127.0.0.1:34157->127.0.0.1:5984: read: connection reset by peer',
     reason: 'could not create connection with the server',
     status: '503' } }
Oops! An error occured.
{ FetchError: {"error":"no_couch","ok":false,"original":"Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/88083af23f948e17d4c73b2b3a5e76b3: read tcp 127.0.0.1:34255->127.0.0.1:5984: read: connection reset by peer","reason":"could not create connection with the server","status":"503"}
  name: 'FetchError',
  response: 
   Body {
     url: 'http://cozy.tools:8080/data/io.cozy.bills/88083af23f948e17d4c73b2b3a5e76b3',
     status: 503,
     statusText: 'Service Unavailable',
     headers: Headers { _headers: [Object] },
     ok: false,
     body: 
      PassThrough {
        _readableState: [Object],
        readable: false,
        domain: null,
        _events: [Object],
        _eventsCount: 4,
        _maxListeners: undefined,
        _writableState: [Object],
        writable: false,
        allowHalfOpen: true,
        _transformState: [Object] },
     bodyUsed: true,
     size: 0,
     timeout: 0,
     _raw: 
      [ <Buffer 7b 22 65 72 72 6f 72 22 3a 22 6e 6f 5f 63 6f 75 63 68 22 2c 22 6f 6b 22 3a 66 61 6c 73 65 2c 22 6f 72 69 67 69 6e 61 6c 22 3a 22 50 75 74 20 68 74 74 ... > ],
     _abort: false,
     _bytes: 305 },
  url: 'http://cozy.tools:8080/data/io.cozy.bills/88083af23f948e17d4c73b2b3a5e76b3',
  status: 503,
  reason: 
   { error: 'no_couch',
     ok: false,
     original: 'Put http://localhost:5984/cozy35ba44d2d1749e6f21646edce51e7190%2Fio-cozy-bills/88083af23f948e17d4c73b2b3a5e76b3: read tcp 127.0.0.1:34255->127.0.0.1:5984: read: connection reset by peer',
     reason: 'could not create connection with the server',
     status: '503' } }

And more.

I have absolutly no idea about the cause. I think Docker and Couch maybe involved. I'm on a pretty decent computer.

ptbrowne commented 6 years ago

Can you share a link to download the JSON ?

LucsT commented 6 years ago

Yes but only for you and in a private way. It's what you want?

doubleface commented 6 years ago

I have the same problem with a remote cozy (a cozy.works). This is related to the number of bills. When there are more than 1000 bills approximately, the this problem happens.

doubleface commented 6 years ago

I suppose we could use bulk commands to be softer on couchdb and also faster