search

danmunn / redmine_dmsf

Fork of svn repository for redmine_dmsf
GNU General Public License v2.0
413 stars 193 forks source link

Cannot open files in utf-8 folders via webdav #299

Closed disjunctor closed 9 years ago

disjunctor commented 9 years ago

When accessing webdav with browser the log contains:

Encoding::CompatibilityError (incompatible character encodings: ASCII-8BIT and UTF-8):
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/resource.rb:390:in `child'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/base_resource.rb:74:in `child'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/dmsf_resource.rb:58:in `block in children'
  .bundle/ruby/2.0.0/gems/activerecord-3.2.17/lib/active_record/relation/delegation.rb:6:in `map'
  .bundle/ruby/2.0.0/gems/activerecord-3.2.17/lib/active_record/relation/delegation.rb:6:in `map'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/dmsf_resource.rb:57:in `children'
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/resource.rb:116:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/resource_proxy.rb:60:in `children'
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/resource.rb:116:in `method_missing'
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/controller.rb:389:in `find_resources'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:72:in `block in propfind'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:96:in `block (2 levels) in render_xml'
  .bundle/ruby/2.0.0/gems/nokogiri-1.6.1/lib/nokogiri/xml/builder.rb:391:in `call'
  .bundle/ruby/2.0.0/gems/nokogiri-1.6.1/lib/nokogiri/xml/builder.rb:391:in `insert'
  .bundle/ruby/2.0.0/gems/nokogiri-1.6.1/lib/nokogiri/xml/builder.rb:375:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:93:in `block in render_xml'
  .bundle/ruby/2.0.0/gems/nokogiri-1.6.1/lib/nokogiri/xml/builder.rb:293:in `initialize'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:92:in `new'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:92:in `render_xml'
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/controller.rb:448:in `multistatus'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/controller.rb:71:in `propfind'
  .bundle/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/handler.rb:30:in `call'
  .bundle/ruby/2.0.0/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  .bundle/ruby/2.0.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  .bundle/ruby/2.0.0/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/routing/route_set.rb:608:in `call'
  .bundle/ruby/2.0.0/gems/rack-openid-1.4.2/lib/rack/openid.rb:98:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/conditionalget.rb:35:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/head.rb:14:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/no_parse.rb:33:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/flash.rb:242:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/cookies.rb:341:in `call'
  .bundle/ruby/2.0.0/gems/activerecord-3.2.17/lib/active_record/query_cache.rb:64:in `call'
  .bundle/ruby/2.0.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `_run__2118164196009012634__call__2189481968152830052__callbacks'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:405:in `__run_callback'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/callbacks.rb:81:in `run_callbacks'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/rack/logger.rb:32:in `call_app'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/rack/logger.rb:16:in `block in call'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:22:in `tagged'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/rack/logger.rb:16:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
  .bundle/ruby/2.0.0/gems/activesupport-3.2.17/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  .bundle/ruby/2.0.0/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
  .bundle/ruby/2.0.0/gems/actionpack-3.2.17/lib/action_dispatch/middleware/static.rb:63:in `call'
  .bundle/ruby/2.0.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  .bundle/ruby/2.0.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  .bundle/ruby/2.0.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  .bundle/ruby/2.0.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  .bundle/ruby/2.0.0/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/engine.rb:484:in `call'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/application.rb:231:in `call'
  .bundle/ruby/2.0.0/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `method_missing'
  .bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:572:in `process_client'
  .bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:666:in `worker_loop'
  .bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:521:in `spawn_missing_workers'
  .bundle/ruby/2.0.0/gems/unicorn-4.8.2/lib/unicorn/http_server.rb:140:in `start'
  .bundle/ruby/2.0.0/gems/unicorn-4.8.2/bin/unicorn_rails:209:in `<top (required)>'
  .bundle/ruby/2.0.0/bin/unicorn_rails:23:in `load'
  .bundle/ruby/2.0.0/bin/unicorn_rails:23:in `<main>'
Environment:
  Redmine version                2.4.5.stable
  Ruby version                   2.0.0-p247 (2013-06-27) [x86_64-linux]
  Rails version                  3.2.17
  Environment                    production
  Database adapter               Mysql2
SCM:
  Git                            1.8.4.5
  Filesystem                     
Redmine plugins:
  redmine_base_select2           0.0.1
  redmine_bootstrap_kit          0.1
  redmine_contacts               3.2.15-light
  redmine_default_custom_query   1.0.0
  redmine_dmsf                   1.4.8 stable
  redmine_git_hosting            0.7.5
  redmine_ics_export             3.0.0.dev
  redmine_latex_mathjax          0.1.0
  redmine_ldap_sync              2.0.3.devel.g7f1c325e31
  redmine_multiprojects_issue    0.1
  redmine_sidekiq                2.0.0
  redmine_theme_changer          0.1.0
  redmine_watcher_groups         0.0.2git
HighwayStar commented 9 years ago

Can be dirty-fixed by two force-encoding patches one in dav4rack gem

diff --git a/resource.rb.bak b/resource.rb
index 4f3e57f..3b6957a 100644
--- a/resource.rb.bak
+++ b/resource.rb
@@ -381,7 +381,7 @@ module DAV4Rack
     # Create a new child with the given name
     # NOTE:: Include trailing '/' if child is collection
     def child(name)
-      new_public = public_path.dup
+      new_public = public_path.dup.force_encoding("utf-8")
       new_public = new_public + '/' unless new_public[-1,1] == '/'
       new_public = '/' + new_public unless new_public[0,1] == '/'
       new_path = path.dup

and one in plugin 

diff --git a/lib/redmine_dmsf/webdav/base_resource.rb b/lib/redmine_dmsf/webdav/base_resource.rb
index 5340c22..74a39d9 100644
--- a/lib/redmine_dmsf/webdav/base_resource.rb
+++ b/lib/redmine_dmsf/webdav/base_resource.rb
@@ -66,6 +66,7 @@ module RedmineDmsf
           '',
           '',
         ] + entities unless parent.nil?
+       entities.force_encoding('utf-8')
         @response.body << index_page % [ path.empty? ? "/" : path, path.empty? ? "/" : path , entities ]
       end

The problem come from bundler/ruby/2.0.0/gems/dav4rack-0.2.11/lib/dav4rack/handler.rb:20 request = Rack::Request.new(env)

after that request.path contains string in Encoding:ASCII-8BIT, when actual data is utf8-path. Inresting that when path contains no utf-8 data this string marked as Encoding:UTF-8

HighwayStar commented 9 years ago

gem rack-utf8_sanitizer also helps

popovse commented 9 years ago

I have similar problem. we have error, when we try to open folders, which have russian names.

Above mentioned tips not helped us.

ralgozino commented 9 years ago

Same problem while trying to upload via WebDAV files with spanish characters like ó, í, ñ, etc.

picman commented 9 years ago

Fixed. Tested with Czech characters: 'Žlutý'. It works as expected.

cityant commented 9 years ago

Have Chinese chars in file/folder name, still failed with the same issue.

picman commented 9 years ago

Do you use 1.5.1-devel?

Can you post a file/folder name in Chinese, that causes problems?

cityant commented 9 years ago

Thanks for your reply. Yes. I have the latest 1.5.1-stable branch.

Environment:
  Redmine version                2.6.3.stable
  Ruby version                   2.1.5-p273 (2014-11-13) [x86_64-linux]
  Rails version                  3.2.21
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.6.11
  Mercurial                      1.4
  Bazaar                         2.1.1
  Git                            1.7.1
  Filesystem                     
Redmine plugins:
  redmine_dmsf                   1.5.1 stable

The the following url causes 500 internal error, that is: http://myserver:9090/dmsf/webdav/test/%E9%A1%B9%E7%9B%AE%E7%AE%A1%E7%90%86

What strange is, there is a file with a Chinese name under the above url/folder which can be OPENED correctly, the link is : http://myserver:9090/dmsf/webdav/test/%E9%A1%B9%E7%9B%AE%E7%AE%A1%E7%90%86/Web%E5%89%8D%E7%AB%AF%E5%BC%80%E5%8F%91%E5%B7%A5%E7%A8%8B%E5%B8%88%E6%8A%80%E8%83%BD.pdf

So the problem might be the index page of the first URL can not be rendered?

Here is the log file:

Connecting to database specified by database.yml
Creating scope :sorted. Overwriting existing method User.sorted.
Creating scope :sorted. Overwriting existing method Group.sorted.
Creating scope :system. Overwriting existing method Enumeration.system.
Starting Time Tracker plugin for RedMine
  UserCustomField Load (0.5ms)  SELECT `custom_fields`.* FROM `custom_fields` WHERE `custom_fields`.`type` IN ('UserCustomField') AND `custom_fields`.`name` = 'Ultraviolet Theme' LIMIT 1
Started GET "/dmsf/webdav/test/%e9%a1%b9%e7%9b%ae%e7%ae%a1%e7%90%86" for ssssss at 2015-04-02 13:06:22 +0800
  Setting Load (0.4ms)  SELECT `settings`.* FROM `settings` WHERE `settings`.`name` = 'plugin_redmine_dmsf' LIMIT 1
  User Load (0.4ms)  SELECT `users`.* FROM `users` WHERE `users`.`type` IN ('User', 'AnonymousUser') AND `users`.`login` = 'xxxx'
  SQL (2.6ms)  UPDATE `users` SET `last_login_on` = '2015-04-02 13:06:22' WHERE `users`.`type` IN ('User', 'AnonymousUser') AND `users`.`id` = 7
  Project Load (0.3ms)  SELECT `projects`.* FROM `projects` WHERE `projects`.`identifier` = 'test' LIMIT 1
  EnabledModule Load (0.4ms)  SELECT `enabled_modules`.* FROM `enabled_modules` WHERE `enabled_modules`.`project_id` = 1
  DmsfFolder Load (0.5ms)  SELECT `dmsf_folders`.* FROM `dmsf_folders` WHERE `dmsf_folders`.`deleted` = 0 AND `dmsf_folders`.`project_id` = 1 AND `dmsf_folders`.`title` = '项目管理' ORDER BY title ASC
  DmsfFolder Load (0.3ms)  SELECT `dmsf_folders`.* FROM `dmsf_folders` WHERE `dmsf_folders`.`dmsf_folder_id` = 11
  DmsfFile Load (0.3ms)  SELECT `dmsf_files`.* FROM `dmsf_files` WHERE `dmsf_files`.`dmsf_folder_id` = 11 AND `dmsf_files`.`deleted` = 0
Encoding::CompatibilityError (incompatible character encodings: ASCII-8BIT and UTF-8):
  dav4rack (0.3.0) lib/dav4rack/resource.rb:406:in `child'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/base_resource.rb:85:in `child'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/dmsf_resource.rb:61:in `block in children'
  activerecord (3.2.21) lib/active_record/relation/delegation.rb:6:in `map'
  activerecord (3.2.21) lib/active_record/relation/delegation.rb:6:in `map'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/dmsf_resource.rb:60:in `children'
  dav4rack (0.3.0) lib/dav4rack/resource.rb:116:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/base_resource.rb:64:in `html_display'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/dmsf_resource.rb:202:in `get'
  dav4rack (0.3.0) lib/dav4rack/resource.rb:116:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/resource_proxy.rb:99:in `get'
  dav4rack (0.3.0) lib/dav4rack/resource.rb:116:in `method_missing'
  dav4rack (0.3.0) lib/dav4rack/controller.rb:78:in `get'
  dav4rack (0.3.0) lib/dav4rack/handler.rb:30:in `call'
  journey (1.0.4) lib/journey/router.rb:68:in `block in call'
  journey (1.0.4) lib/journey/router.rb:56:in `each'
  journey (1.0.4) lib/journey/router.rb:56:in `call'
  actionpack (3.2.21) lib/action_dispatch/routing/route_set.rb:608:in `call'
  rack-openid (1.4.2) lib/rack/openid.rb:98:in `call'
  request_store (1.0.5) lib/request_store/middleware.rb:9:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  rack (1.4.5) lib/rack/etag.rb:23:in `call'
  rack (1.4.5) lib/rack/conditionalget.rb:25:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/head.rb:14:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/no_parse.rb:33:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/flash.rb:242:in `call'
  rack (1.4.5) lib/rack/session/abstract/id.rb:210:in `context'
  rack (1.4.5) lib/rack/session/abstract/id.rb:205:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/cookies.rb:341:in `call'
  activerecord (3.2.21) lib/active_record/query_cache.rb:64:in `call'
  activerecord (3.2.21) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  activesupport (3.2.21) lib/active_support/callbacks.rb:405:in `_run__1925844483086144649__call__3523164737377064723__callbacks'
  activesupport (3.2.21) lib/active_support/callbacks.rb:405:in `__run_callback'
  activesupport (3.2.21) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  activesupport (3.2.21) lib/active_support/callbacks.rb:81:in `run_callbacks'
  actionpack (3.2.21) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  railties (3.2.21) lib/rails/rack/logger.rb:32:in `call_app'
  railties (3.2.21) lib/rails/rack/logger.rb:18:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/request_id.rb:22:in `call'
  rack (1.4.5) lib/rack/methodoverride.rb:21:in `call'
  rack (1.4.5) lib/rack/runtime.rb:17:in `call'
  activesupport (3.2.21) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  rack (1.4.5) lib/rack/lock.rb:15:in `call'
  actionpack (3.2.21) lib/action_dispatch/middleware/static.rb:83:in `call'
  rack-cache (1.2) lib/rack/cache/context.rb:136:in `forward'
  rack-cache (1.2) lib/rack/cache/context.rb:245:in `fetch'
  rack-cache (1.2) lib/rack/cache/context.rb:185:in `lookup'
  rack-cache (1.2) lib/rack/cache/context.rb:66:in `call!'
  rack-cache (1.2) lib/rack/cache/context.rb:51:in `call'
  railties (3.2.21) lib/rails/engine.rb:484:in `call'
  railties (3.2.21) lib/rails/application.rb:231:in `call'
  railties (3.2.21) lib/rails/railtie/configurable.rb:30:in `method_missing'
  passenger (5.0.1) lib/phusion_passenger/rack/thread_handler_extension.rb:85:in `process_request'
  passenger (5.0.1) lib/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
  passenger (5.0.1) lib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
  passenger (5.0.1) lib/phusion_passenger/request_handler.rb:414:in `block (3 levels) in start_threads'
  passenger (5.0.1) lib/phusion_passenger/utils.rb:111:in `block in create_thread_and_abort_on_exception'

The line 406 in dav4rack (0.3.0) lib/dav4rack/resource.rb is: self.class.new("#{new_public}#{name}", "#{new_path}#{name}", request, response, options.merge(:user => @user))

picman commented 9 years ago

Fixed. chinese_chars_in_path

cityant commented 9 years ago

Thanks for the fix! But in master branch, the line 81 (if parent?) in lib/redmine_dmsf/webdav/base_resource.rb seems to cause recession in my site. The original removal line 79 (unless parent.nil?) just works after changing back.

Log error:

...
NoMethodError (Undefined method: parent? for class #<RedmineDmsf::Webdav::ProjectResource:0x007fe0bd206160>.):
  dav4rack (0.3.0) lib/dav4rack/resource.rb:114:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/base_resource.rb:81:in `html_display'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/project_resource.rb:87:in `get'
  ...
cityant commented 9 years ago

Got an new error after applying the new change :s

NoMethodError (Undefined method: parent? for class < RedmineDmsf::Webdav::IndexResource:0x007f16615371a8 >.):
  dav4rack (0.3.0) lib/dav4rack/resource.rb:114:in `method_missing'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/base_resource.rb:81:in `html_display'
  plugins/redmine_dmsf/lib/redmine_dmsf/webdav/index_resource.rb:79:in `get'
cityant commented 9 years ago

Thanks for your quick fix! It works!