search

radical-collaboration / extasy-grlsd

Repository to hold the input data and scripts for the ExTASY gromacs-lsdmap work
1 stars 1 forks source link

quota exceeded #102

Closed euhruska closed 6 years ago

euhruska commented 6 years ago

Is this quota exceeded or something else?

2018-10-21 13:43:00,556: radical.entk.task_manager.0000: task-manager                    : MainThread     : DEBUG   : task.0049 with state SUBMITTING synced with AppManager
2018-10-21 13:43:00,557: radical.entk.task_manager.0000: task-manager                    : MainThread     : INFO    : Transition of task.0049 to new state SUBMITTING successful
2018-10-21 13:43:00,558: radical.entk.task_manager.0000: task-manager                    : MainThread     : INFO    : Received heartbeat request
2018-10-21 13:43:00,558: radical.entk.task_manager.0000: task-manager                    : MainThread     : INFO    : Sent heartbeat response
2018-10-21 13:43:01,310: radical.entk.task_manager.0000: task-manager                    : MainThread     : ERROR   : Error in task execution: pymongo error: {'nModified': 0, 'nUpserted': 0, 'nMatched': 0, 'writeErrors': [{u'index': 29, u'code': 12$
01, u'errmsg': u'quota exceeded', u'op': {'resource_sandbox': None, 'control': 'umgr', 'uid': 'unit.000029', 'stdout': None, '_id': 'unit.000029', 'states': ['NEW'], 'name': 'task.0029,None,stage.0000,None,pipeline.0000,None', 'client_sandbox': Non$
, 'umgr': 'umgr.0000', 'description': {'kernel': '', 'cpu_thread_type': '', 'post_exec': [], 'gpu_process_type': '', 'executable': 'bwpy-environ', 'stdout': '', 'pre_exec': ['module unload PrgEnv-cray', 'module load PrgEnv-gnu', 'module unload gcc'$
 'module load gcc/5.3.0', 'module unload bwpy', 'module load bwpy/2.0.0-pre0', 'module load bwpy-mpi', 'module add cudatoolkit', 'export MODULEPATH="/sw/bw/bwpy/modulefiles/:${MODULEPATH}"', 'export CPATH="${BWPY_INCLUDE_PATH}"', 'export LDFLAGS="$$
LDFLAGS} -Wl,--rpath=${BWPY_LIBRARY_PATH}"', 'module load craype-ml-plugin-py3/1.1.0', 'export MPICH_GNI_MALLOC_FALLBACK=enable', 'export MPICH_GNI_MAX_VSHORT_MSG_SIZE=64', 'export MPICH_MAX_THREAD_SAFETY=multiple', 'export MPICH_RMA_OVER_DMAPP=1',
'export OPENMM_PLUGIN_DIR=/mnt/bwpy/single/usr/lib/plugins', 'source /projects/sciteam/bamm/hruska/vpy8/bin/activate', 'printenv > env.log', 'export OMP_NUM_THREADS=1', 'export PYEMMA_NJOBS=1', 'export tasks=md', 'export iter=1'], 'environment': {}$
 'cleanup': False, 'arguments': ['python', 'run_openmm.py', '--trajstride', '10000', '--Kconfig', 'settings_extasy_tica3_chignolin_long.wcfg', '--idxstart', '27', '--idxend', '28', '--path', '/u/sciteam/hruska/scratch/extasy_tica3_chignolin_long', $
--iter', '1', '--md_steps', '10000000', '--save_traj', 'True', '>', 'md.log'], 'gpu_processes': 1, 'cpu_processes': 0, 'restartable': False, 'output_staging': [{'uid': 'sd.0157', 'priority': 0, 'source': 'md.log', 'flags': 64, 'action': 'Copy', 'ta$
get': '/u/sciteam/hruska/scratch/extasy_tica3_chignolin_long/md_logs/iter1_md27.log'}], 'gpu_thread_type': '', 'cpu_threads': 0, 'cpu_process_type': '', 'pilot': '', 'name': 'task.0029,None,stage.0000,None,pipeline.0000,None', 'input_staging': [{'u$
d': 'sd.0153', 'priority': 0, 'source': 'pilot:///run-openmm-xml3.py', 'flags': 64, 'action': 'Link', 'target': 'run_openmm.py'}, {'uid': 'sd.0154', 'priority': 0, 'source': 'pilot:///system-5.xml', 'flags': 64, 'action': 'Link', 'target': 'system-$
.xml'}, {'uid': 'sd.0155', 'priority': 0, 'source': 'pilot:///integrator-5.xml', 'flags': 64, 'action': 'Link', 'target': 'integrator-5.xml'}, {'uid': 'sd.0156', 'priority': 0, 'source': 'pilot:///settings_extasy_tica3_chignolin_long.wcfg', 'flags'$
 64, 'action': 'Link', 'target': 'settings_extasy_tica3_chignolin_long.wcfg'}], 'stderr': '', 'gpu_threads': 1}, 'cmd': [], 'exit_code': None, 'state': 'NEW', 'stderr': None, 'pilot': '', 'type': 'unit', 'unit_sandbox': None, 'pilot_sandbox': None}$
], 'upserted': [], 'writeConcernErrors': [], 'nRemoved': 0, 'nInserted': 29}
Traceback (most recent call last):
  File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/execman/rp/task_manager.py", line 226, in _tmgr
    umgr.submit_units(bulk_cuds)
  File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 743, in submit_units
    self._session._dbs.insert_units(unit_docs)
  File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/pilot/db/database.py", line 406, in insert_units
    raise RuntimeError( 'pymongo error: %s' % e.details)
RuntimeError: pymongo error: {'nModified': 0, 'nUpserted': 0, 'nMatched': 0, 'writeErrors': [{u'index': 29, u'code': 12501, u'errmsg': u'quota exceeded', u'op': {'resource_sandbox': None, 'control': 'umgr', 'uid': 'unit.000029', 'stdout': None, '_i$
': 'unit.000029', 'states': ['NEW'], 'name': 'task.0029,None,stage.0000,None,pipeline.0000,None', 'client_sandbox': None, 'umgr': 'umgr.0000', 'description': {'kernel': '', 'cpu_thread_type': '', 'post_exec': [], 'gpu_process_type': '', 'executable$
: 'bwpy-environ', 'stdout': '', 'pre_exec': ['module unload PrgEnv-cray', 'module load PrgEnv-gnu', 'module unload gcc', 'module load gcc/5.3.0', 'module unload bwpy', 'module load bwpy/2.0.0-pre0', 'module load bwpy-mpi', 'module add cudatoolkit',
'export MODULEPATH="/sw/bw/bwpy/modulefiles/:${MODULEPATH}"', 'export CPATH="${BWPY_INCLUDE_PATH}"', 'export LDFLAGS="${LDFLAGS} -Wl,--rpath=${BWPY_LIBRARY_PATH}"', 'module load craype-ml-plugin-py3/1.1.0', 'export MPICH_GNI_MALLOC_FALLBACK=enable'$
 'export MPICH_GNI_MAX_VSHORT_MSG_SIZE=64', 'export MPICH_MAX_THREAD_SAFETY=multiple', 'export MPICH_RMA_OVER_DMAPP=1', 'export OPENMM_PLUGIN_DIR=/mnt/bwpy/single/usr/lib/plugins', 'source /projects/sciteam/bamm/hruska/vpy8/bin/activate', 'printenv
> env.log', 'export OMP_NUM_THREADS=1', 'export PYEMMA_NJOBS=1', 'export tasks=md', 'export iter=1'], 'environment': {}, 'cleanup': False, 'arguments': ['python', 'run_openmm.py', '--trajstride', '10000', '--Kconfig', 'settings_extasy_tica3_chignol$
n_long.wcfg', '--idxstart', '27', '--idxend', '28', '--path', '/u/sciteam/hruska/scratch/extasy_tica3_chignolin_long', '--iter', '1', '--md_steps', '10000000', '--save_traj', 'True', '>', 'md.log'], 'gpu_processes': 1, 'cpu_processes': 0, 'restarta$
le': False, 'output_staging': [{'uid': 'sd.0157', 'priority': 0, 'source': 'md.log', 'flags': 64, 'action': 'Copy', 'target': '/u/sciteam/hruska/scratch/extasy_tica3_chignolin_long/md_logs/iter1_md27.log'}], 'gpu_thread_type': '', 'cpu_threads': 0,
'cpu_process_type': '', 'pilot': '', 'name': 'task.0029,None,stage.0000,None,pipeline.0000,None', 'input_staging': [{'uid': 'sd.0153', 'priority': 0, 'source': 'pilot:///run-openmm-xml3.py', 'flags': 64, 'action': 'Link', 'target': 'run_openmm.py'}$
 {'uid': 'sd.0154', 'priority': 0, 'source': 'pilot:///system-5.xml', 'flags': 64, 'action': 'Link', 'target': 'system-5.xml'}, {'uid': 'sd.0155', 'priority': 0, 'source': 'pilot:///integrator-5.xml', 'flags': 64, 'action': 'Link', 'target': 'integ$
ator-5.xml'}, {'uid': 'sd.0156', 'priority': 0, 'source': 'pilot:///settings_extasy_tica3_chignolin_long.wcfg', 'flags': 64, 'action': 'Link', 'target': 'settings_extasy_tica3_chignolin_long.wcfg'}], 'stderr': '', 'gpu_threads': 1}, 'cmd': [], 'exi$
_code': None, 'state': 'NEW', 'stderr': None, 'pilot': '', 'type': 'unit', 'unit_sandbox': None, 'pilot_sandbox': None}}], 'upserted': [], 'writeConcernErrors': [], 'nRemoved': 0, 'nInserted': 29}
Process task-manager:
andre-merzky commented 6 years ago

Yes, indeed, quota exceeded. What MongoDB are you using?

euhruska commented 6 years ago

mlab.com, I'm below 1GB, so not sure what quota is exceeded

vivek-bala commented 6 years ago

I think unless you have a paid mlab db instance, the default is 240MB. You might have to create a new instance if you are using the free db instance.