search

dagster-io / fake-star-detector

https://github.com/dagster-io/dagster
234 stars 19 forks source link

Docker version fails after two hour run - out of memory? #5

Closed kgolubic closed 1 year ago

kgolubic commented 1 year ago

I was using simpler_model on repo with 10k+ stars. After 8800 stars, I got an error message. Is there a way to increase memory that can be used?

Multiprocess executor: child process for step stargazers_with_user_info was terminated by signal 9 (SIGKILL). This usually indicates that the process was killed by the operating system due to running out of memory. Possible solutions include increasing the amount of memory available to the run, reducing the amount of memory used by the ops in the run, or configuring the executor to run fewer ops concurrently.
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 240, in execute
    event_or_none = next(step_iter)
,  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 357, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 240, in execute
    event_or_none = next(step_iter)
,  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 357, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)
Dependencies for step classified_stargazers_df failed: ['stargazers_with_user_info']. Not executing.

Dependencies for step real_vs_raw_stars_report were not executed: ['classified_stargazers_df']. Not executing.

Dependencies for step github_stars_notebook_gist were not executed: ['real_vs_raw_stars_report']. Not executing.

image

abhishek9sharma commented 1 year ago

If you are using Docker desktop, you can assign more RAM to Docker (and containers) using the steps mentioned at https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container

kgolubic commented 1 year ago

Thanks! I've used the method described at https://fizzylogic.nl/2023/01/05/how-to-configure-memory-limits-in-wsl2. Closing the issue.