search

mikeal / node.couchapp.js

Utility for writing couchapps.
Apache License 2.0
406 stars 78 forks source link

couchapp opening too many files. #59

Open boxxxie opened 12 years ago

boxxxie commented 12 years ago

read below...

boxxxie commented 12 years ago

actually, i failed to reproduce this. seems like a situational bug. however there is a bug related to uploading, as i'm pushing 20k files, and only 500 are being uploaded. report back when i figure out what is wrong.

boxxxie commented 12 years ago

so. it looks like i'm uploading too many files (20k).

err { [Error: EMFILE, open '/####/cxml/allDocuments.1_files/images/FOPL-1_files/4']
  errno: 20,
  code: 'EMFILE',
  path: '/#####/cxml/allDocuments.1_files/images/FOPL-1_files/4' }

http://stackoverflow.com/questions/10355501/connect-emfile-error-in-node-js

boxxxie commented 12 years ago

following this thread about ulimit in ubuntu http://www.ubun2.com/question/433/how_set_ulimit_ubuntu_linux_getting_sudo_ulimit_command_not_found_error i set my ulimit to 100000 couchapp still gives me an error:

err { [Error: EISDIR, read] errno: 28, code: 'EISDIR' }

it's one error instead of 20k errors, so big improvement. however i think that this program should not need to open so many files at once.

rjcorwin commented 12 years ago

I'm experiencing a similar symptom but I'm not seeing any error from the couchapp utility or on the couchdb console. The design document ends up on the couch, but alas, many attachments are missing.

couchapp: https://github.com/rawberg/Hub-List_GTD-Productivity OS: Mac OS X

boxxxie commented 12 years ago

i was only able to see the errors because i edited the couchapp program and added logging to it.

boxxxie commented 11 years ago

anyone else having this issue may find that my (in progress) bash port of node.couchapp could be useful. https://github.com/boxxxie/bash.couchapp

rjcorwin commented 10 years ago

@mikeal I'll have to abandon node.couchapp.js for all my projects if we can't figure out how to fix this bug :-(

rjcorwin commented 10 years ago

@boxxxie You are onto something. I'm on a Mac so I searched around for this ulimit thing in relation to Mac OS. Ran across the following thread on StackOverflow.

How to change default ulimit values in Mac OS X 10.6?

The problem statement starts out, "Mac OS X limits number of processes to 266 and open files to 256. Sometimes this is problem; 256 open files and 266 processes aren't that much."

This sounds like node.couchapp.js will have to batch the files opened to get around this problem on all systems.