ruby-concurrency / concurrent-ruby

Modern concurrency tools including agents, futures, promises, thread pools, supervisors, and more. Inspired by Erlang, Clojure, Scala, Go, Java, JavaScript, and classic concurrency patterns.
https://ruby-concurrency.github.io/concurrent-ruby/
Other
5.68k stars 418 forks source link

Concurrent::Promise causing error with Rails.cache/.fetch #957

Closed cwli24 closed 2 years ago

cwli24 commented 2 years ago
    #Make a HTTP GET promise for each tag (execute inline).
    (0...tags.size).each do |idx|
      fetch_requests[idx] = Concurrent::Promise.execute do
        # We'll read from the cache first to see if the data already lives there.
        response = Rails.cache.fetch( tags[idx], expires_in: 3.hours, race_condition_ttl: 2) do
          HTTP.get(EXTDATA_URL, :params => {:tag => tags[idx]})
        end
         ...
     Concurrent::Promise.all?(*fetch_requests).execute.wait!

This is causing the same crash from the server. I tried using :file_store as cache rather than :memory_store (which was made 'thread-safe' by Rails anyways), and also using ReentrantReadWrite lock to gate the cache, but no success. This is what's showing:

image

* Operating system:                linux
* Ruby implementation:             Ruby
* `concurrent-ruby` version:       1.1.10
* `concurrent-ruby-ext` installed: yes
* `concurrent-ruby-edge` used:     no
cwli24 commented 2 years ago

Actually this may be(?) related to an issue with the cache; just the trace has a poor way of showing it. I tried using the native Thread class via replacing Promise.execute with Thread.new and fetch_requests.each(&:join), and it's also failing on the fetch return, saving data to the cache. I'm suspicious the cache might just not like storing http responses..

cwli24 commented 2 years ago

I'm fairly certain the root of this is actually caused by what's being stored into the cache, and subsequently this problem. I'm going to close this issue, and link the one I opened on Rails' github.