search

estately / rets

A pure-ruby library for fetching data from RETS servers
https://github.com/estately/rets
127 stars 94 forks source link

ArgumentError: marshal data too short #212

Closed jondavidchristopher closed 6 years ago

jondavidchristopher commented 7 years ago

I have been getting random "ArgumentError: marshal data too short" in our logs for some time now. I would assume that somehow the file cache is being erased or corrupted on Heroku post/mid restart. This is not the most optimal fix and I am open to other suggestions. Stack trace below:

…ms/rets-0.10.0/lib/rets/metadata/
marshal_serializer.rb:  19:in `load'
…by/2.3.0/gems/rets-0.10.0/lib/rets/metadata/
caching.rb:  43:in `block in load'
…2.3.0/gems/rets-0.10.0/lib/rets/metadata/
file_cache.rb:  22:in `open'
…2.3.0/gems/rets-0.10.0/lib/rets/metadata/
file_cache.rb:  22:in `load'
…by/2.3.0/gems/rets-0.10.0/lib/rets/metadata/
caching.rb:  42:in `load'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb: 245:in `metadata'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb: 153:in `find_rets_class'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb: 144:in `find_every'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb:  88:in `find_with_given_retry'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb:  83:in `find_with_retries'
…/bundle/ruby/2.3.0/gems/rets-0.10.0/lib/rets/
client.rb:  74:in `find'
                          /app/app/classes/mls/
fetch.rb:  14:in `get_records'
                         /app/app/classes/mls/
import.rb:  18:in `store_media_records'
                          /app/app/jobs/
mls_keys_job.rb:  14:in `perform'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb: 152:in `execute_job'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb: 134:in `block (2 levels) in process'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 128:in `block in invoke'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…gems/sidekiq-pro-3.3.3/lib/sidekiq/batch/
middleware.rb:  26:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…2.3.0/gems/sidekiq-ent-1.3.2/lib/sidekiq-ent/
unique.rb: 124:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…idekiq-ent-1.3.2/lib/sidekiq-ent/limiter/
middleware.rb:  38:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…q-4.1.4/lib/sidekiq/middleware/server/
active_record.rb:   6:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…ekiq-4.1.4/lib/sidekiq/middleware/server/
retry_jobs.rb:  74:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…sidekiq-4.1.4/lib/sidekiq/middleware/server/
logging.rb:  11:in `block in call'
…e/ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
logging.rb:  32:in `with_context'
…sidekiq-4.1.4/lib/sidekiq/middleware/server/
logging.rb:   7:in `call'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 130:in `block in invoke'
…3.0/gems/sidekiq-4.1.4/lib/sidekiq/middleware/
chain.rb: 133:in `invoke'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb: 129:in `block in process'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb: 168:in `stats'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb: 128:in `process'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb:  80:in `process_one'
…ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
processor.rb:  68:in `run'
…ndle/ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
util.rb:  17:in `watchdog'
…ndle/ruby/2.3.0/gems/sidekiq-4.1.4/lib/sidekiq/
util.rb:  25:in `block in safe_thread'
dougcole commented 7 years ago

I suspect the root of the problem has to do with using the Rets::Metadata::FileCache on Heroku. I would replace that with something that doesn't try and write to the filesystem (which is discourage on Heroku, I believe). FileCache is just meant as an example, you can write a similar class to write to the database for example.