AlchemyCMS / alchemy-dragonfly-s3

Store AlchemyCMS files on Amazon S3
https://alchemy-cms.com
MIT License
1 stars 7 forks source link

Errors when using different data stores based on the environment #10

Closed afdev82 closed 3 years ago

afdev82 commented 3 years ago

At first I included this gem for all environment in my Gemfile in order to use the S3 storage on Heroku. Before I was using the file data store for all my environment, but since Heroku doesn't support it, I had to use S3 in production. I'm using AlchemyCMS v4.6.2 and for that reason I used the branch 'alchemy-4'. Dragonfly v1.3.0 and Rails v6.0.3.6. If I use only the s3 data store in the dragonfly configuration, everything is fine. But in this way I'm forced to use s3 also in development, I wanted to use my local filesystem instead (like before the change). So I reverted the dragonfly configuration like it was before, eg:

Dragonfly.app(:alchemy_pictures).configure do
  dragonfly_url nil
  plugin :imagemagick
  plugin :svg
  secret xxx
  url_format '/pictures/:job/:name.:ext'

  datastore :file,
            root_path: Rails.root.join('uploads', 'pictures').to_s,
            server_root: Rails.root.join('public'),
            store_meta: false
end

After this change I get the following error if I try to load a page:

Dragonfly::FileDataStore::UnableToFormUrl in Alchemy::Pages#index 

couldn't form url for uid "2021/01/19/5gidig7ks1_show.jpeg" with root_path "/home/antonio/Projects/SmartMirror/Server/uploads/pictures" and server_root "/home/antonio/Projects/SmartMirror/Server/public"

Full Stacktrace:
dragonfly (1.3.0) lib/dragonfly/file_data_store.rb:144:in `url_for'
dragonfly (1.3.0) lib/dragonfly/app.rb:240:in `remote_url_for'
dragonfly (1.3.0) lib/dragonfly/model/attachment.rb:98:in `remote_url'
alchemy-dragonfly-s3 (a83d745dab79) app/models/alchemy/picture/s3_url.rb:17:in `call'
alchemy-dragonfly-s3 (a83d745dab79) lib/alchemy/picture_monkey_patch.rb:25:in `url'
app/views/alchemy/elements/_landing_section.html.erb:18:in `block (2 levels) in _1f89752fd4afe04dd4ff0290af59e667'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_block_helper.rb:124:in `block in element_view_for'
actionview (6.0.3.4) lib/action_view/helpers/capture_helper.rb:45:in `block in capture'
actionview (6.0.3.4) lib/action_view/helpers/capture_helper.rb:209:in `with_output_buffer'
actionview (6.0.3.4) lib/action_view/helpers/capture_helper.rb:45:in `capture'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_block_helper.rb:123:in `element_view_for'
app/views/alchemy/elements/_landing_section.html.erb:8:in `block in _1f89752fd4afe04dd4ff0290af59e667'
actionview (6.0.3.4) lib/action_view/helpers/cache_helper.rb:171:in `cache'
app/views/alchemy/elements/_landing_section.html.erb:7:in `_1f89752fd4afe04dd4ff0290af59e667'
actionview (6.0.3.4) lib/action_view/base.rb:274:in `_run'
actionview (6.0.3.4) lib/action_view/template.rb:185:in `block in render'
activesupport (6.0.3.4) lib/active_support/notifications.rb:182:in `instrument'
actionview (6.0.3.4) lib/action_view/template.rb:385:in `instrument_render_template'
actionview (6.0.3.4) lib/action_view/template.rb:183:in `render'
deface (1.5.3) lib/deface/action_view_extensions.rb:43:in `render'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:357:in `block in render_partial'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:346:in `render_partial'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:317:in `render'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:65:in `render_partial_to_object'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:53:in `render_partial'
actionview (6.0.3.4) lib/action_view/helpers/rendering_helper.rb:38:in `render'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_helper.rb:182:in `render_element'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_helper.rb:113:in `block in render_elements'
activerecord (6.0.3.4) lib/active_record/relation/delegation.rb:87:in `each'
activerecord (6.0.3.4) lib/active_record/relation/delegation.rb:87:in `each'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_helper.rb:112:in `each_with_index'
alchemy_cms (4.6.2) app/helpers/alchemy/elements_helper.rb:112:in `render_elements'
app/views/alchemy/page_layouts/_landing.html.erb:17:in `_0e2ccc9673b3a832ff798cf3dde4f4ee'
actionview (6.0.3.4) lib/action_view/base.rb:274:in `_run'
actionview (6.0.3.4) lib/action_view/template.rb:185:in `block in render'
activesupport (6.0.3.4) lib/active_support/notifications.rb:182:in `instrument'
actionview (6.0.3.4) lib/action_view/template.rb:385:in `instrument_render_template'
actionview (6.0.3.4) lib/action_view/template.rb:183:in `render'
deface (1.5.3) lib/deface/action_view_extensions.rb:43:in `render'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:357:in `block in render_partial'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:346:in `render_partial'
actionview (6.0.3.4) lib/action_view/renderer/partial_renderer.rb:317:in `render'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:65:in `render_partial_to_object'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:53:in `render_partial'
actionview (6.0.3.4) lib/action_view/helpers/rendering_helper.rb:38:in `render'
alchemy_cms (4.6.2) app/helpers/alchemy/pages_helper.rb:50:in `render_page_layout'
alchemy_cms (4.6.2) app/views/alchemy/pages/show.html.erb:1:in `_1c71d7bb85b1fcd1d8fee420edfabffe'
actionview (6.0.3.4) lib/action_view/base.rb:274:in `_run'
actionview (6.0.3.4) lib/action_view/template.rb:185:in `block in render'
activesupport (6.0.3.4) lib/active_support/notifications.rb:182:in `instrument'
actionview (6.0.3.4) lib/action_view/template.rb:385:in `instrument_render_template'
actionview (6.0.3.4) lib/action_view/template.rb:183:in `render'
deface (1.5.3) lib/deface/action_view_extensions.rb:43:in `render'
actionview (6.0.3.4) lib/action_view/renderer/template_renderer.rb:58:in `block (2 levels) in render_template'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:88:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/abstract_renderer.rb:87:in `instrument'
actionview (6.0.3.4) lib/action_view/renderer/template_renderer.rb:57:in `block in render_template'
actionview (6.0.3.4) lib/action_view/renderer/template_renderer.rb:65:in `render_with_layout'
actionview (6.0.3.4) lib/action_view/renderer/template_renderer.rb:56:in `render_template'
solidus_multi_domain (516b1ac140eb) app/decorators/lib/solidus_multi_domain/action_view/template_renderer_decorator.rb:9:in `render_template'
solidus_multi_domain (516b1ac140eb) app/decorators/lib/solidus_multi_domain/action_view/template_renderer_decorator.rb:9:in `render_template'
actionview (6.0.3.4) lib/action_view/renderer/template_renderer.rb:13:in `render'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
actionview (6.0.3.4) lib/action_view/renderer/renderer.rb:29:in `render_to_object'
actionview (6.0.3.4) lib/action_view/rendering.rb:117:in `block in _render_template'
actionview (6.0.3.4) lib/action_view/base.rb:304:in `in_rendering_context'
actionview (6.0.3.4) lib/action_view/rendering.rb:116:in `_render_template'
actionpack (6.0.3.4) lib/action_controller/metal/streaming.rb:218:in `_render_template'
actionview (6.0.3.4) lib/action_view/rendering.rb:103:in `render_to_body'
actionpack (6.0.3.4) lib/action_controller/metal/rendering.rb:52:in `render_to_body'
actionpack (6.0.3.4) lib/action_controller/metal/renderers.rb:142:in `render_to_body'
actionpack (6.0.3.4) lib/abstract_controller/rendering.rb:25:in `render'
actionpack (6.0.3.4) lib/action_controller/metal/rendering.rb:36:in `render'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
activesupport (6.0.3.4) lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/home/antonio/.rbenv/versions/2.7.2/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
activesupport (6.0.3.4) lib/active_support/core_ext/benchmark.rb:14:in `ms'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:44:in `block in render'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
activerecord (6.0.3.4) lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:43:in `render'
alchemy_cms (4.6.2) app/controllers/alchemy/pages_controller.rb:144:in `block (2 levels) in render_page'
actionpack (6.0.3.4) lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
alchemy_cms (4.6.2) app/controllers/alchemy/pages_controller.rb:142:in `render_page'
alchemy_cms (4.6.2) app/controllers/alchemy/pages_controller.rb:73:in `show'
alchemy_cms (4.6.2) app/controllers/alchemy/pages_controller.rb:54:in `index'
actionpack (6.0.3.4) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (6.0.3.4) lib/abstract_controller/base.rb:195:in `process_action'
actionpack (6.0.3.4) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (6.0.3.4) lib/abstract_controller/callbacks.rb:42:in `block in process_action'
activesupport (6.0.3.4) lib/active_support/callbacks.rb:112:in `block in run_callbacks'
activesupport (6.0.3.4) lib/active_support/core_ext/time/zones.rb:66:in `use_zone'
app/controllers/application_controller.rb:81:in `use_time_zone'
activesupport (6.0.3.4) lib/active_support/callbacks.rb:121:in `block in run_callbacks'
react-rails (2.6.1) lib/react/rails/controller_lifecycle.rb:31:in `use_react_component_helper'
activesupport (6.0.3.4) lib/active_support/callbacks.rb:121:in `block in run_callbacks'
activesupport (6.0.3.4) lib/active_support/callbacks.rb:139:in `run_callbacks'
actionpack (6.0.3.4) lib/abstract_controller/callbacks.rb:41:in `process_action'
actionpack (6.0.3.4) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `block in instrument'
activesupport (6.0.3.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.0.3.4) lib/active_support/notifications.rb:180:in `instrument'
actionpack (6.0.3.4) lib/action_controller/metal/instrumentation.rb:32:in `process_action'
actionpack (6.0.3.4) lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
activerecord (6.0.3.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (6.0.3.4) lib/abstract_controller/base.rb:136:in `process'
actionview (6.0.3.4) lib/action_view/rendering.rb:39:in `process'
actionpack (6.0.3.4) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.0.3.4) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.0.3.4) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.0.3.4) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:49:in `block in serve'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.0.3.4) lib/action_dispatch/routing/route_set.rb:834:in `call'
railties (6.0.3.4) lib/rails/engine.rb:527:in `call'
railties (6.0.3.4) lib/rails/railtie.rb:190:in `public_send'
railties (6.0.3.4) lib/rails/railtie.rb:190:in `method_missing'
actionpack (6.0.3.4) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
actionpack (6.0.3.4) lib/action_dispatch/routing/mapper.rb:48:in `serve'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:49:in `block in serve'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (6.0.3.4) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.0.3.4) lib/action_dispatch/routing/route_set.rb:834:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
bullet (6.1.3) lib/bullet/rack.rb:12:in `call'
http_accept_language (2.1.1) lib/http_accept_language/middleware.rb:14:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
omniauth (1.9.1) lib/omniauth/strategy.rb:192:in `call!'
omniauth (1.9.1) lib/omniauth/strategy.rb:169:in `call'
omniauth (1.9.1) lib/omniauth/builder.rb:45:in `call'
dragonfly (1.3.0) lib/dragonfly/middleware.rb:14:in `call'
dragonfly (1.3.0) lib/dragonfly/middleware.rb:14:in `call'
rack-attack (6.4.0) lib/rack/attack.rb:97:in `call'
rack-attack (6.4.0) lib/rack/attack.rb:104:in `call'
warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
warden (1.2.9) lib/warden/manager.rb:34:in `catch'
warden (1.2.9) lib/warden/manager.rb:34:in `call'
rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.3) lib/rack/etag.rb:27:in `call'
rack (2.2.3) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.3) lib/rack/head.rb:12:in `call'
actionpack (6.0.3.4) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/cookies.rb:648:in `call'
activerecord (6.0.3.4) lib/active_record/migration.rb:567:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.0.3.4) lib/active_support/callbacks.rb:101:in `run_callbacks'
actionpack (6.0.3.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
web-console (4.1.0) lib/web_console/middleware.rb:132:in `call_app'
web-console (4.1.0) lib/web_console/middleware.rb:28:in `block in call'
web-console (4.1.0) lib/web_console/middleware.rb:17:in `catch'
web-console (4.1.0) lib/web_console/middleware.rb:17:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
railties (6.0.3.4) lib/rails/rack/logger.rb:37:in `call_app'
railties (6.0.3.4) lib/rails/rack/logger.rb:26:in `block in call'
activesupport (6.0.3.4) lib/active_support/tagged_logging.rb:80:in `block in tagged'
activesupport (6.0.3.4) lib/active_support/tagged_logging.rb:28:in `tagged'
activesupport (6.0.3.4) lib/active_support/tagged_logging.rb:80:in `tagged'
railties (6.0.3.4) lib/rails/rack/logger.rb:26:in `call'
sprockets-rails (3.2.2) lib/sprockets/rails/quiet_assets.rb:13:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
request_store (1.5.0) lib/request_store/middleware.rb:19:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/request_id.rb:27:in `call'
rack (2.2.3) lib/rack/method_override.rb:24:in `call'
rack (2.2.3) lib/rack/runtime.rb:22:in `call'
activesupport (6.0.3.4) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
dragonfly (1.3.0) lib/dragonfly/cookie_monster.rb:9:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/static.rb:126:in `call'
rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
actionpack (6.0.3.4) lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
rack-cors (1.1.1) lib/rack/cors.rb:100:in `call'
webpacker (5.2.1) lib/webpacker/dev_server_proxy.rb:25:in `perform_request'
rack-proxy (0.6.5) lib/rack/proxy.rb:57:in `call'
railties (6.0.3.4) lib/rails/engine.rb:527:in `call'
passenger (6.0.6) src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
passenger (6.0.6) src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
passenger (6.0.6) src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
passenger (6.0.6) src/ruby_supportlib/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'
passenger (6.0.6) src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

If I move this gem only in the group for the production / staging environments, the error is gone. I think this is a bug, because it should be possible to do something like that in the dragonfly configuration to use different datastores based on the environment:

Dragonfly.app(:alchemy_pictures).configure do
  dragonfly_url nil

  plugin :imagemagick
  plugin :svg

  secret xxx
  url_format '/pictures/:job/:name.:ext'

  if Rails.env.production? || Rails.env.staging?
    datastore :s3,
              bucket_name: ENV['AWS_S3_BUCKET_NAME'],
              access_key_id: ENV['AWS_ACCESS_KEY_ID'],
              secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
              region: ENV['AWS_REGION'],
              root_path: "#{Rails.env}/alchemy/pictures",
              url_host: ENV['AWS_S3_BUCKET_NAME']
  else
    datastore :file,
              root_path: Rails.root.join('uploads', 'pictures').to_s,
              server_root: Rails.root.join('public'),
              store_meta: false
  end
end

Then I face another problem if I try to get the url for a video essence, not strictly related with this issue, but I'm wondering if there is a unified way to get the url of the resource in all environment, considering the different configuration of dragonfly. Using the s3 datastore I had to change the way to get the url for the video in the view from

alchemy.show_attachment_path(
  landing_video_source.ingredient(:source),
  format: landing_video_source.ingredient(:source).suffix
),

to

landing_video_source.ingredient(:source).url

Because Safari won't play the video when the byte range requests are not supported (eg. on Heroku web server). Now in development I have to use the first code and in production the second one, otherwise I get:

undefined method `url' for #<Alchemy::Attachment:0x00007f3cb1eb4d68>

I saw that the url method is defined in this gem here. It would be very ugly to have a conditional statement based on the environment to choose at runtime what to use in the view, but at the moment is the only solution that I can see. Something like that:

Rails.application.config.x.use_s3_storage ? landing_video_source.ingredient(:source).url : alchemy.show_attachment_path(
  landing_video_source.ingredient(:source),
  format: landing_video_source.ingredient(:source).suffix
)

From my point of view, there is no way to use my local filesystem in development and s3 in production, only one datastore for both. But it sounds very strange to me, are there possible solutions to this problem?

afdev82 commented 3 years ago

Maybe a possible solution would be to identify the datastore in use and only if the datastore is s3 replace the classes in this block:

https://github.com/AlchemyCMS/alchemy-dragonfly-s3/blob/main/lib/alchemy/dragonfly/s3/engine.rb#L9

tvdeyen commented 3 years ago

This gem is meant to be used with a s3 setup. If you want to use it only in production, then the gem needs to be in the production gem group and you need to adjust your app to reflect that. There are many advantages in using this gem and s3 in development as well, but this is totally up to you and not the responsibility of this gem.

Regarding the missing methods (url and remote_url): These methods are defined by the datastore and are not available without it. You can either this datastore in development as well or use respond_to? or is_a? to check if they are available.

afdev82 commented 3 years ago

OK, got it. Thank you.