search

fog / fog-aws

Module for the 'fog' gem to support Amazon Web Services http://aws.amazon.com/
MIT License
300 stars 353 forks source link

NoMethodError (undefined method `copy_object' for "close":String) #702

Open softmonkeyjapan opened 8 months ago

softmonkeyjapan commented 8 months ago

Since today on my production server, I'm experiencing a weird error that seems to come from fog-was's copy_object method call. I've checked other library invovled and none of them have a method name copy_object.

Stack

When doing a regular file assignment:

object.image = File.open(...)
object.save!

I'm getting the following error:

NoMethodError (undefined method `copy_object' for "close":String)

If I try to do the same with remote_attribute_url I still get the error. Not sure if this is a carrierwave issue or not. Anyone can help ?

EDIT

Not sure if this is related but whenever I'm affecting a file to my model uploader, i'm getting the following warning:

[fog][DEPRECATION] #connection= is deprecated, pass :service in at creation (/Users/loic/.rbenv/versions/2.7.0/lib/ruby/gems/2.7.0/gems/fog-core-2.3.0/lib/fog/core/attributes.rb:134:in `block in merge_attributes')
geemus commented 8 months ago

Hey, sorry to hear you are having issues. I haven't heard of this before, so I'm left to speculate a bit.

This sounds like a type-safety kind of issue. copy_object is certainly defined on fog's File objects, so my initial guess is that somewhere that expects a file object is being given a string (or perhaps a non-fog File) and then the fog and/or carrierwave machinery is trying to do it's thing and is not succeeding.

I don't know enough about your particular setup to pin this down just yet. Could you share a bit more of what the code in object looks like?

Also, I don't think that deprecation should lead to this particular problem, though it would be good to fix that issue as well (as eventually I'd like to remove the deprecated things altogether to clean things up).

iprog21 commented 5 months ago

Is this resolved? I'm facing the same issue.

geemus commented 5 months ago

@iprog21 I hadn't heard any additional details, so I didn't have anything else to go on. I'm happy to look again and try to work through it with you if you can provide some more details of how you are using things and maybe some stack traces.

iprog21 commented 5 months ago

@iprog21 I hadn't heard any additional details, so I didn't have anything else to go on. I'm happy to look again and try to work through it with you if you can provide some more details of how you are using things and maybe some stack traces.

HI @geemus . I saw the root in my code, seems its on the conversion of the image when uploading. its on the ImageMagick. My temporary fix is disable the versioning since this is only my personal project so I am aware to upload small size image only.

So if you can check it too, try adding versions, if that will appear on your end.

geemus commented 5 months ago

Glad to hear you have a workaround for now. Unfortunately on my side I haven't used these libraries myself in quite some time, so I don't have an existing setup that I can easily change to test against.

sectasy0 commented 4 months ago

Hello yall, I have similar problem, exact error while using CarrierWave with MiniMagic and I can't use workaround @iprog21 described, any ideas? I'm using Ruby 3.3

gem 'rails', '~> 7.1.3.2'
gem 'carrierwave', '~> 3.0'
gem 'fog-aws', '~> 3.22.0'
sectasy0 commented 4 months ago 
undefined method `copy_object' for an instance of String (NoMethodError)

            service.copy_object(directory.key, key, target_directory_key, target_file_key, options)

Any ideas @geemus?

geemus commented 4 months ago

@sectasy0 Hmm, it looks like you expect service to be a fog storage service object, but that it's actually a string. How are you setting the service value and/or where are you getting it from? I think that might help narrow it down.

iprog21 commented 4 months ago

Hello yall, I have similar problem, exact error while using CarrierWave with MiniMagic and I can't use workaround @iprog21 described, any ideas? I'm using Ruby 3.3

gem 'rails', '~> 7.1.3.2'
gem 'carrierwave', '~> 3.0'
gem 'fog-aws', '~> 3.22.0'

Hi @sectasy0 . In my situation, I fix it by removing versions(:thumb, :medium, etc..) on the uploader file. I don't why is that happening. So you might want to limit the uploading size to prevent large file.

sectasy0 commented 4 months ago

@sectasy0 Hmm, it looks like you expect service to be a fog storage service object, but that it's actually a string. How are you setting the service value and/or where are you getting it from? I think that might help narrow it down.

Could you please clarify your question? I encountered this error while trying to save an object using. I configured fog to use DigitalOcean spaces with cdn.

geemus commented 4 months ago

@sectasy0 I'll try.

As an example, I'd expect the error you are seeing from something like this:

service = "example-string"
service.copy_object(directory.key, key, target_directory_key, target_file_key, options)

Really service should be more like:

service = Fog::AWS::Storage.new(...)

Does that make more sense?

sectasy0 commented 4 months ago

@geemus Thank you for you fast response, but I'm not creating the service from my code, I'm using Carrierwave uploader and this error not happens every time, one of my users is experiences strange problems after trying to add record with images. Do you thing its more CarrierWave issue? I was trying to search internet but i found nothing

geemus commented 4 months ago

@sectasy0 could you share some of the code from the uploader to show how you are using it around where the error occurs?

sectasy0 commented 4 months ago

@geemus Sure

module Product
  class ImageUploader < CarrierWave::Uploader::Base
    include CarrierWave::MiniMagick

    @fog_public = true

    storage :fog
    process resize_to_fit: [800, 400]

    version :thumb do
      process resize_to_fill: [200, 200]
    end

    def store_dir
      "uploads/products/#{model.id}"
    end

    def size_range
      0..(1.megabytes)
    end

    def content_type_allowlist
      [%r{image/}]
    end

    def extension_allowlist
      %w[jpg jpeg png gif]
    end

    def filename
      "image.#{file&.extension}" if file
    end
  end
end
geemus commented 4 months ago

@sectasy0 thanks, could you share the stacktrace from the error as well? I think that will help pin down where the issue might be coming from.

sectasy0 commented 4 months ago

@geemus Here is full stack trace:

carrierwave (3.0.7) lib/carrierwave/storage/fog.rb in copy_to at line 444
carrierwave (3.0.7) lib/carrierwave/storage/fog.rb in store at line 330
carrierwave (3.0.7) lib/carrierwave/storage/fog.rb in store! at line 86
carrierwave (3.0.7) lib/carrierwave/uploader/store.rb in block in store! at line 83
carrierwave (3.0.7) lib/carrierwave/uploader/callbacks.rb in with_callbacks at line 15
carrierwave (3.0.7) lib/carrierwave/uploader/store.rb in store! at line 82
carrierwave (3.0.7) lib/carrierwave/uploader/versions.rb in block in store_versions! at line 333
carrierwave (3.0.7) lib/carrierwave/uploader/versions.rb in each_value at line 333
carrierwave (3.0.7) lib/carrierwave/uploader/versions.rb in store_versions! at line 333
carrierwave (3.0.7) lib/carrierwave/uploader/callbacks.rb in block in with_callbacks at line 16
carrierwave (3.0.7) lib/carrierwave/uploader/callbacks.rb in each at line 16
carrierwave (3.0.7) lib/carrierwave/uploader/callbacks.rb in with_callbacks at line 16
carrierwave (3.0.7) lib/carrierwave/uploader/store.rb in store! at line 82
carrierwave (3.0.7) lib/carrierwave/mounter.rb in each at line 142
carrierwave (3.0.7) lib/carrierwave/mounter.rb in store! at line 142
carrierwave (3.0.7) lib/carrierwave/mount.rb in store_file! at line 368
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in make_lambda at line 403
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in halting_and_conditional at line 239
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in invoke_after at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in each at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in invoke_after at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in run_callbacks at line 111
activesupport (7.1.3.3) lib/active_support/callbacks.rb in _run_save_callbacks at line 952
activerecord (7.1.3.3) lib/active_record/callbacks.rb in create_or_update at line 441
activerecord (7.1.3.3) lib/active_record/timestamp.rb in create_or_update at line 125
activerecord (7.1.3.3) lib/active_record/persistence.rb in save at line 718
activerecord (7.1.3.3) lib/active_record/validations.rb in save at line 49
activerecord (7.1.3.3) lib/active_record/transactions.rb in block in save at line 309
activerecord (7.1.3.3) lib/active_record/transactions.rb in block in with_transaction_returning_status at line 365
activerecord (7.1.3.3) lib/active_record/connection_adapters/abstract/database_statements.rb in transaction at line 342
activerecord (7.1.3.3) lib/active_record/transactions.rb in with_transaction_returning_status at line 361
activerecord (7.1.3.3) lib/active_record/transactions.rb in save at line 309
activerecord (7.1.3.3) lib/active_record/suppressor.rb in save at line 52
activerecord (7.1.3.3) lib/active_record/associations/collection_association.rb in insert_record at line 373
activerecord (7.1.3.3) lib/active_record/associations/has_many_association.rb in insert_record at line 63
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in block in save_collection_association at line 415
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in each at line 406
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in save_collection_association at line 406
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in block in add_autosave_association_callbacks at line 193
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in instance_eval at line 166
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in block in define_non_cyclic_method at line 166
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in make_lambda at line 403
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in halting_and_conditional at line 239
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in invoke_after at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in each at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in invoke_after at line 602
activesupport (7.1.3.3) lib/active_support/callbacks.rb in run_callbacks at line 111
activesupport (7.1.3.3) lib/active_support/callbacks.rb in _run_create_callbacks at line 952
activerecord (7.1.3.3) lib/active_record/callbacks.rb in _create_record at line 445
activerecord (7.1.3.3) lib/active_record/timestamp.rb in _create_record at line 114
activerecord (7.1.3.3) lib/active_record/persistence.rb in create_or_update at line 1221
activerecord (7.1.3.3) lib/active_record/callbacks.rb in block in create_or_update at line 441
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 121
activerecord (7.1.3.3) lib/active_record/autosave_association.rb in around_save_collection_association at line 375
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 130
activesupport (7.1.3.3) lib/active_support/callbacks.rb in run_callbacks at line 141
activesupport (7.1.3.3) lib/active_support/callbacks.rb in _run_save_callbacks at line 952
activerecord (7.1.3.3) lib/active_record/callbacks.rb in create_or_update at line 441
activerecord (7.1.3.3) lib/active_record/timestamp.rb in create_or_update at line 125
activerecord (7.1.3.3) lib/active_record/persistence.rb in save at line 718
activerecord (7.1.3.3) lib/active_record/validations.rb in save at line 49
activerecord (7.1.3.3) lib/active_record/transactions.rb in block in save at line 309
activerecord (7.1.3.3) lib/active_record/transactions.rb in block in with_transaction_returning_status at line 365
activerecord (7.1.3.3) lib/active_record/connection_adapters/abstract/transaction.rb in block in within_new_transaction at line 535
activesupport (7.1.3.3) lib/active_support/concurrency/null_lock.rb in synchronize at line 9
activerecord (7.1.3.3) lib/active_record/connection_adapters/abstract/transaction.rb in within_new_transaction at line 532
activerecord (7.1.3.3) lib/active_record/connection_adapters/abstract/database_statements.rb in transaction at line 344
activerecord (7.1.3.3) lib/active_record/transactions.rb in with_transaction_returning_status at line 361
activerecord (7.1.3.3) lib/active_record/transactions.rb in save at line 309
activerecord (7.1.3.3) lib/active_record/suppressor.rb in save at line 52
globalize (6.3.0) lib/globalize/active_record/instance_methods.rb in block (2 levels) in save at line 166
globalize (6.3.0) lib/globalize/active_record/instance_methods.rb in without_fallbacks at line 271
globalize (6.3.0) lib/globalize/active_record/instance_methods.rb in block in save at line 165
globalize (6.3.0) lib/globalize.rb in with_locale at line 37
globalize (6.3.0) lib/globalize/active_record/instance_methods.rb in save at line 164
app/controllers/admin/products_controller.rb in create at line 42
In App

    # POST /admin/products/
    def create
      @product = Product::Base.new(product_params)
      if @product.save
        redirect_to(admin_products_path, notice: I18n.t('admin.products.created'))
      else
        render(:new, status: :unprocessable_entity)
Called from: actionpack (7.1.3.3) lib/action_controller/metal/basic_implicit_render.rb in send_action

Hide 89 more frames
actionpack (7.1.3.3) lib/abstract_controller/base.rb in process_action at line 224
actionpack (7.1.3.3) lib/action_controller/metal/rendering.rb in process_action at line 165
actionpack (7.1.3.3) lib/abstract_controller/callbacks.rb in block in process_action at line 259
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 121
turbo-rails (2.0.5) lib/turbo-rails.rb in with_request_id at line 24
turbo-rails (2.0.5) app/controllers/concerns/turbo/request_id_tracking.rb in turbo_tracking_request_id at line 10
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 130
actiontext (7.1.3.3) lib/action_text/rendering.rb in with_renderer at line 23
actiontext (7.1.3.3) lib/action_text/engine.rb in block (4 levels) in <class:Engine> at line 69
activesupport (7.1.3.3) lib/active_support/callbacks.rb in instance_exec at line 130
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 130
sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb in block in sentry_around_action at line 17
sentry-ruby (5.17.3) lib/sentry/hub.rb in block in with_child_span at line 109
sentry-ruby (5.17.3) lib/sentry/span.rb in with_child_span at line 211
sentry-ruby (5.17.3) lib/sentry/hub.rb in with_child_span at line 107
sentry-ruby (5.17.3) lib/sentry-ruby.rb in with_child_span at line 490
sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb in sentry_around_action at line 14
activesupport (7.1.3.3) lib/active_support/callbacks.rb in block in run_callbacks at line 130
activesupport (7.1.3.3) lib/active_support/callbacks.rb in run_callbacks at line 141
actionpack (7.1.3.3) lib/abstract_controller/callbacks.rb in process_action at line 258
actionpack (7.1.3.3) lib/action_controller/metal/rescue.rb in process_action at line 25
actionpack (7.1.3.3) lib/action_controller/metal/instrumentation.rb in block in process_action at line 74
activesupport (7.1.3.3) lib/active_support/notifications.rb in block in instrument at line 206
activesupport (7.1.3.3) lib/active_support/notifications/instrumenter.rb in instrument at line 58
sentry-rails (5.17.3) lib/sentry/rails/tracing.rb in instrument at line 54
activesupport (7.1.3.3) lib/active_support/notifications.rb in instrument at line 206
actionpack (7.1.3.3) lib/action_controller/metal/instrumentation.rb in process_action at line 73
actionpack (7.1.3.3) lib/action_controller/metal/params_wrapper.rb in process_action at line 261
activerecord (7.1.3.3) lib/active_record/railties/controller_runtime.rb in process_action at line 32
actionpack (7.1.3.3) lib/abstract_controller/base.rb in process at line 160
actionview (7.1.3.3) lib/action_view/rendering.rb in process at line 40
actionpack (7.1.3.3) lib/action_controller/metal.rb in dispatch at line 227
actionpack (7.1.3.3) lib/action_controller/metal.rb in dispatch at line 309
actionpack (7.1.3.3) lib/action_dispatch/routing/route_set.rb in dispatch at line 49
actionpack (7.1.3.3) lib/action_dispatch/routing/route_set.rb in serve at line 32
actionpack (7.1.3.3) lib/action_dispatch/journey/router.rb in block in serve at line 51
actionpack (7.1.3.3) lib/action_dispatch/journey/router.rb in block in find_routes at line 131
actionpack (7.1.3.3) lib/action_dispatch/journey/router.rb in each at line 124
actionpack (7.1.3.3) lib/action_dispatch/journey/router.rb in find_routes at line 124
actionpack (7.1.3.3) lib/action_dispatch/journey/router.rb in serve at line 32
actionpack (7.1.3.3) lib/action_dispatch/routing/route_set.rb in call at line 882
rack (3.0.11) lib/rack/deflater.rb in call at line 47
warden (1.2.9) lib/warden/manager.rb in block in call at line 36
warden (1.2.9) lib/warden/manager.rb in catch at line 34
warden (1.2.9) lib/warden/manager.rb in call at line 34
rack (3.0.11) lib/rack/tempfile_reaper.rb in call at line 20
rack (3.0.11) lib/rack/etag.rb in call at line 29
rack (3.0.11) lib/rack/conditional_get.rb in call at line 43
rack (3.0.11) lib/rack/head.rb in call at line 15
actionpack (7.1.3.3) lib/action_dispatch/http/permissions_policy.rb in call at line 36
actionpack (7.1.3.3) lib/action_dispatch/http/content_security_policy.rb in call at line 33
rack-session (2.0.0) lib/rack/session/abstract/id.rb in context at line 272
rack-session (2.0.0) lib/rack/session/abstract/id.rb in call at line 266
actionpack (7.1.3.3) lib/action_dispatch/middleware/cookies.rb in call at line 689
actionpack (7.1.3.3) lib/action_dispatch/middleware/callbacks.rb in block in call at line 29
activesupport (7.1.3.3) lib/active_support/callbacks.rb in run_callbacks at line 101
actionpack (7.1.3.3) lib/action_dispatch/middleware/callbacks.rb in call at line 28
sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb in call at line 12
actionpack (7.1.3.3) lib/action_dispatch/middleware/debug_exceptions.rb in call at line 29
sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb in block (2 levels) in call at line 29
sentry-ruby (5.17.3) lib/sentry/hub.rb in with_session_tracking at line 251
sentry-ruby (5.17.3) lib/sentry-ruby.rb in with_session_tracking at line 403
sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb in block in call at line 20
sentry-ruby (5.17.3) lib/sentry/hub.rb in with_scope at line 59
sentry-ruby (5.17.3) lib/sentry-ruby.rb in with_scope at line 383
sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb in call at line 19
actionpack (7.1.3.3) lib/action_dispatch/middleware/show_exceptions.rb in call at line 31
railties (7.1.3.3) lib/rails/rack/logger.rb in call_app at line 37
railties (7.1.3.3) lib/rails/rack/logger.rb in block in call at line 24
activesupport (7.1.3.3) lib/active_support/tagged_logging.rb in block in tagged at line 135
activesupport (7.1.3.3) lib/active_support/tagged_logging.rb in tagged at line 39
activesupport (7.1.3.3) lib/active_support/tagged_logging.rb in tagged at line 135
activesupport (7.1.3.3) lib/active_support/broadcast_logger.rb in method_missing at line 240
railties (7.1.3.3) lib/rails/rack/logger.rb in call at line 24
actionpack (7.1.3.3) lib/action_dispatch/middleware/remote_ip.rb in call at line 92
app/middlewares/cloudflare.rb in call at line 19
1
In App

        env['REMOTE_ADDR'] = env['HTTP_CF_CONNECTING_IP']
        env['HTTP_X_FORWARDED_FOR'] = env['HTTP_CF_CONNECTING_IP']
      end
      @app.call(env)
    end
  end
end
actionpack (7.1.3.3) lib/action_dispatch/middleware/request_id.rb in call at line 28
1
rack (3.0.11) lib/rack/runtime.rb in call at line 24
actionpack (7.1.3.3) lib/action_dispatch/middleware/executor.rb in call at line 14
actionpack (7.1.3.3) lib/action_dispatch/middleware/static.rb in call at line 25
rack (3.0.11) lib/rack/sendfile.rb in call at line 114
actionpack (7.1.3.3) lib/action_dispatch/middleware/host_authorization.rb in call at line 141
railties (7.1.3.3) lib/rails/engine.rb in call at line 536
puma (6.4.2) lib/puma/configuration.rb in call at line 272
puma (6.4.2) lib/puma/request.rb in block in handle_request at line 100
puma (6.4.2) lib/puma/thread_pool.rb in with_force_shutdown at line 378
puma (6.4.2) lib/puma/request.rb in handle_request at line 99
puma (6.4.2) lib/puma/server.rb in process_client at line 464
puma (6.4.2) lib/puma/server.rb in block in run at line 245
sectasy0 commented 4 months ago

Hello @geemus, Have you been able to determine what the cause is?

geemus commented 4 months ago

Thanks for sharing further details, I hadn't had a chance to look over the weekend but have reviewed now. Unfortunately it's not obvious to me from reviewing this where the error is occurring, it seems as though it may be in the internals of carrierwave, rather than in fog or in your usage (though I could be missing something still). I think you may be better served by reporting this against the carrierwave repository and hopefully someone with more familiarity there can be of more direct help.

sectasy0 commented 3 months ago

Hello again @geemus - Could you take look at this? I think it might be cause of this problem it occurs along with this problem undefined method copy_object

[fog][DEPRECATION] #connection= is deprecated, pass :service in at creation (bundle/ruby/3.3.0/gems/fog-core-2.4.0/lib/fog/core/attributes.rb:146:inblock in merge_attributes')`

geemus commented 3 months ago

I dug around some more and think I've narrowed it down a little at least, but I'm still not sure and have a couple additional questions when you get a chance.

Could you share what the credentials/configuration you are passing into carrierwave for fog looks like? Please omit the actual credentials, really I mostly need to see the keys I think.

Also, do some carrierwave things work and some fail, or do they all fail?

sectasy0 commented 3 months ago

@geemus Thank you for fast response. Normally it works and its really rare but yesterday I was uploading around 960 images and 50 of them fails with this error I've send yesterday.

config/initializers/carrierwave.rb

CarrierWave.configure do |config|
  config.fog_provider = 'fog/aws'
  config.fog_credentials = {
    provider: 'AWS',
    aws_access_key_id: ENV.fetch('DO_KEY', Rails.application.credentials[:do_key]),
    aws_secret_access_key: ENV.fetch('DO_SECRET', Rails.application.credentials[:do_secret]),
    region: ENV.fetch('DO_REGION', Rails.application.credentials[:do_region]),
    endpoint: 'https://fra1.digitaloceanspaces.com'
  }
  config.fog_directory  = ENV.fetch('DO_BUCKET', Rails.application.credentials[:do_bucket])
  config.fog_attributes = { 'Cache-Control' => 'max-age=315576000' }
  config.fog_public = false
  config.asset_host = 'https://test.fra1.cdn.digitaloceanspaces.com'
  config.cache_dir = "#{Rails.root}/tmp/uploads"
end
geemus commented 3 months ago

Glad to here it's at least mostly working, though it may also say something about why we are having trouble figuring out exactly what is going on. That config looks ok to me.

The signs continue to make me think this is more in carrierwave than fog, though that may not be the most helpful yet.

I have a couple more questions to hopefully continue to narrow in on this.

Have you noticed anything different about the jobs that fail vs the ones that succeed? Are you able to retry the failed jobs? If so, do they succeed or fail on retry?

Thanks for continuing to work through it with me. I wish it were simpler and fixed, but it's turning out to be a real challenge to pin down so far.

sectasy0 commented 3 months ago

Glad to here it's at least mostly working, though it may also say something about why we are having trouble figuring out exactly what is going on. That config looks ok to me.

The signs continue to make me think this is more in carrierwave than fog, though that may not be the most helpful yet.

I have a couple more questions to hopefully continue to narrow in on this.

Have you noticed anything different about the jobs that fail vs the ones that succeed? Are you able to retry the failed jobs? If so, do they succeed or fail on retry?

Thanks for continuing to work through it with me. I wish it were simpler and fixed, but it's turning out to be a real challenge to pin down so far.

The jobs are the same and after retry its okay. Now I'm uploading again above 1k of images and after uploading above 300 images it occured only three times, so I assume its completely random.

geemus commented 2 months ago

@sectasy0 K, thanks. I'll try to find some time to look through it some more soon.

mshibuya commented 1 month ago

Hello from CarrierWave 👋 I think I found something possibly related.

On storing a file into the bucket, http header values in the response will be merged into Fog::AWS::Storage::File's attributes.
https://github.com/fog/fog-aws/blob/1f38afb4daec60f71f88e0c4ba056d9e646cad9d/lib/fog/aws/models/storage/file.rb#L284 What if the DigitalOcean S3-compatible storage occasionally returns the header Connection: close? Then it gets merged into the File model's attribute, triggering:

[fog][DEPRECATION] #connection= is deprecated, pass :service in at creation (`bundle/ruby/3.3.0/gems/fog-core-2.4.0/lib/fog/core/attributes.rb:146:in block in merge_attributes')

And subsequent call to #copy_object will fail with:

NoMethodError (undefined method `copy_object' for "close":String)

I don't have the access to the DigitalOcean storage so it's not confirmed yet, but this can be something that explains what's happening. If true, we'll need to exclude Connection header like this:

           else
             data = service.put_object(directory.key, key, body, options)
-            merge_attributes(data.headers.reject {|key, value| ['Content-Length', 'Content-Type'].include?(key)})
+            merge_attributes(data.headers.reject {|key, value| ['Content-Length', 'Content-Type', 'Connection'].include?(key)})
           end
geemus commented 1 month ago

@mshibuya - Thanks for sharing this, that all sounds very plausible. I suspect it could be any of the providers also, not just DigitalOcean, but it's hard to know for sure. In any event, it seems like a safe thing to try. Thanks!

geemus commented 1 month ago

I've created a PR #725 with this change.

I will plan on trying to do a release with that change in the next few days, and will post here when I do. Hopefully then we can try things with this change and confirm if the issues is still happening or if it has resolved.

geemus commented 1 month ago

Released in v3.27.0.

@sectasy0 hopefully this will resolve it for you, I'll be curious to hear when you get a chance to try it.

sectasy0 commented 1 month ago

@geemus Hello, good to hear! I'll test it ASAP.