issues when running on macOs

[ArglosStudios]

Hello, I am running a project with solid_queue.

When doing some development on my windows pc with WSL, solid_queue runs greatly! However, when running on my macbook, I get either get file descriptors or weird reading errors:

Bad file descriptors:

10:16:17 jobs.1 | [SolidQueue] Restarting fork[5568] (status: 1)
10:16:17 jobs.1 | [SolidQueue] Starting Worker(pid=5592, hostname=Carloss-MacBook-Pro.local, metadata={:polling_interval=>0.3, :queues=>"*", :thread_pool_size=>4})
10:16:48 jobs.1 | /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/interruptible.rb:21:in `wait_readable': Bad file descriptor (Errno::EBADF)
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/interruptible.rb:21:in `interruptible_sleep'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/worker.rb:29:in `run'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:48:in `block (2 levels) in do_start_loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.3/lib/active_support/execution_wrapper.rb:92:in `wrap'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/app_executor.rb:7:in `wrap_in_app_executor'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:47:in `block in do_start_loop'
10:16:48 jobs.1 |       from <internal:kernel>:187:in `loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:44:in `do_start_loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:39:in `start_loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:15:in `start'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:123:in `block in start_fork'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:122:in `fork'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:122:in `start_fork'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:152:in `replace_fork'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:134:in `block in reap_and_replace_terminated_forks'
10:16:48 jobs.1 |       from <internal:kernel>:187:in `loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:130:in `reap_and_replace_terminated_forks'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:51:in `block in supervise'
10:16:48 jobs.1 |       from <internal:kernel>:187:in `loop'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:47:in `supervise'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:26:in `start'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/supervisor.rb:14:in `start'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/tasks.rb:4:in `block (2 levels) in <main>'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/bin/rake:25:in `load'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/bin/rake:25:in `<top (required)>'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `load'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `kernel_load'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:23:in `run'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli.rb:451:in `exec'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli.rb:34:in `dispatch'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/cli.rb:28:in `start'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/3.3.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/bin/bundle:25:in `load'
10:16:48 jobs.1 |       from /Users/carlos/.rbenv/versions/3.3.0/bin/bundle:25:in `<main>'
10:16:49 jobs.1 | [SolidQueue] Restarting fork[5592] (status: 1)

end of file reached error:

10:25:29 jobs.1 | <internal:io>:63:in `read_nonblock': end of file reached (EOFError)
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/interruptible.rb:22:in `block in interruptible_sleep'
10:25:29 jobs.1 |   from <internal:kernel>:187:in `loop'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/interruptible.rb:22:in `interruptible_sleep'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/worker.rb:29:in `run'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:48:in `block (2 levels) in do_start_loop'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.3/lib/active_support/execution_wrapper.rb:92:in `wrap'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/app_executor.rb:7:in `wrap_in_app_executor'
10:25:29 jobs.1 |   from /Users/carlos/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/solid_queue-0.2.2/lib/solid_queue/processes/runnable.rb:47:in `block in do_start_loop'

[ArglosStudios]

this happened both when running via the puma plugin or starting solid_queue via Procfile.

It still run jobs if I enqueue them. However the worker crashes often.

Workaround so far is to run inside docker

[natannobre]

I think I have the same issue on my mac. A piece of the message is solid_queue | /Users/natannobrechaves/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/pg-1.5.4/lib/pg/connection.rb:690: [BUG] Segmentation fault at 0x000000011d5b0b0e.

I'm running inside Procfile.dev.

[34code]

using rosetta to run your terminal (iterm for me) also seems to do the trick!

https://apple.stackexchange.com/questions/428768/on-apple-m1-with-rosetta-how-to-open-entire-terminal-iterm-in-x86-64-architec

[34code]

nvm.. pulled trigger too early. fails with rosetta as well

[feapaydin]

@natannobre I'm also having the same issue. Seems like it has been solved earlier here: https://github.com/rails/solid_queue/issues/145#issuecomment-1954024164

[rosa]

Hey @ArglosStudios, is this still happening for you? Did https://github.com/rails/solid_queue/issues/145#issuecomment-1954024164 help at all?

[rjhancock]

I have this issue but with a MySQL backend (I don't have the option to swap it for another DB).

[rosa]

Hey @rjhancock, does this still happen for you? Are you running the last version?

[rjhancock]

Just tested with latest gems for everything and this is the error I get.

[rosa]

Thanks a lot for testing 🙏 This one seems to be similar or the same as https://github.com/brianmario/mysql2/issues/1163 🤔 It seems it affects Rails's parallel testing too: https://github.com/rails/rails/issues/41176

There's a suggested solution here: compile mysql2 with a MySQL 5.7 client. Do you think this could work for you?

[rjhancock]

Sadly no as deployed MySQL is 8.0 same with local MySQL via Homebrew.

[rosa]

That's ok! You can continue using MySQL 8.0, this is just for the mysql2 gem. You can compile it against a MySQL 5.7 client and continue using it to access a MySQL 8.0 server.

This is what @johnmaxwell proposed here:

gem uninstall mysql2
brew install mysql-client@5.7
gem install mysql2 -- --with-mysql-dir="/usr/local/opt/mysql-client@5.7/"

[rjhancock]

When I test my Solid Queue code, I just run it within Docker Compose. Not ideal but works. I'd rather not add another monkey patch to the instructions and code base for a bug in one of two libraries that would bring my development environment more out of line from the production environment.

[rosa]

That makes sense! I wonder why you get the error then 🤔 In theory, this happens only in macOS, so I assumed you were running everything in your local machine.

[rjhancock]

Only happens on macOS. When running via Docker Compose, runs fine. The servers are all Ubuntu boxes running dockerized containers with Ruby 3.3.4 as the base image.

I try not to run the images inside Docker while working on the code as it greatly slows things down over all. I generally don't need to test the Job system as it's already well tested.

Most of the code I can run without working with queues so it doesn't impact too much. Just wish the bug could be found as looking through the solutions it looks like this has been a bug somewhere for the last several years.

[34code]

I recently switched to Omakub and ubuntu for development due to a number of macos related issues after the ARM on mac transition for this exact reason.. Like you, I run on production using kamal and docker containers (on x86_64 hw) so being on ubuntu helps me more accurately replicate production. Would be nice to identify a root cause on mac though.