search

sup-heliotrope / sup

A curses threads-with-tags style email client (mailing list: supmua@googlegroups.com)
http://sup-heliotrope.github.io
GNU General Public License v2.0
889 stars 96 forks source link

Error from psych after sup-config #577

Closed krerkkiat closed 2 years ago

krerkkiat commented 4 years ago

exception-log.txt content

--- Psych::SyntaxError from thread: main
(/home/krerkkiat/.sup/sources.yaml): did not find expected whitespace or line break while scanning a tag at line 2 column 3
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse_stream'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:390:in `parse'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:277:in `load'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:578:in `block in load_file'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `open'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `load_file'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:145:in `load_yaml_obj'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:227:in `load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/index.rb:102:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:146:in `<module:Redwood>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:76:in `<top (required)>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `<main>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'

content of .sup/sources.yml

---
- !supmua.org,2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/mail/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []
IPv2 commented 4 years ago

Thanks for reporting @krerkkiat.

I haven't managed to reproduce this yet - your .sup/sources.yaml is accepted by psych (and so also by sup) when I tried on my machine.

Please could you try running these two commands and let me know the full output for both? These should provide some extra diagnostic information.

ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'

hexdump -C /home/krerkkiat/.sup/sources.yaml

krerkkiat commented 4 years ago 
$ hexdump -C /home/krerkkiat/.sup/sources.yaml
00000000  25 59 41 4d 4c 20 31 2e  31 0a 2d 2d 2d 0a 2d 20  |%YAML 1.1.---.- |
00000010  21 6d 61 73 61 6e 6a 69  6e 2e 6e 65 74 2c 32 30  |!masanjin.net,20|
00000020  30 36 2d 31 30 2d 30 31  2f 52 65 64 77 6f 6f 64  |06-10-01/Redwood|
00000030  2f 4d 61 69 6c 64 69 72  0a 20 20 75 72 69 3a 20  |/Maildir.  uri: |
00000040  6d 61 69 6c 64 69 72 3a  2f 68 6f 6d 65 2f 6b 72  |maildir:/home/kr|
00000050  65 72 6b 6b 69 61 74 2f  70 65 72 73 6f 6e 61 6c  |erkkiat/personal|
00000060  0a 20 20 75 73 75 61 6c  3a 20 74 72 75 65 0a 20  |.  usual: true. |
00000070  20 61 72 63 68 69 76 65  64 3a 20 66 61 6c 73 65  | archived: false|
00000080  0a 20 20 69 64 3a 20 31  0a 20 20 6c 61 62 65 6c  |.  id: 1.  label|
00000090  73 3a 20 5b 5d 0a                                 |s: [].|
00000096
$ ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'
2.7.0
Traceback (most recent call last):
    7: from -e:1:in `<main>'
    6: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `load_file'
    5: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:577:in `open'
    4: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:578:in `block in load_file'
    3: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:277:in `load'
    2: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:390:in `parse'
    1: from /home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse_stream'
/home/krerkkiat/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/psych.rb:456:in `parse': (/home/krerkkiat/.sup/sources.yaml): did not find expected whitespace or line break while scanning a tag at line 2 column 3 (Psych::SyntaxError)

The same error occurs even with ruby 2.6.5.

However, if I modify the sources.yaml to be

---
- !<masanjin.net,2006-10-01/Redwood/Maildir>
  uri: maildir:/home/krerkkiat/personal
  usual: true
  archived: false
  id: 1
  labels: []

It works.

$ ruby -e 'require "yaml"; puts RUBY_VERSION; puts YAML::load_file "/home/krerkkiat/.sup/sources.yaml"'
2.7.0
{"uri"=>"maildir:/home/krerkkiat/personal", "usual"=>true, "archived"=>false, "id"=>1, "labels"=>[]}

Does it have to do with changing spec in YAML? This comment on libyaml mentioned that comma is invalid in tags. I did try to specify the version to be 1.1, but that still produce the same error. The libyaml on my machine is 0.2.5.

Although, if I use <> in the tag, I got this error instead.

$ sup-config
Traceback (most recent call last):
    11: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
    10: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
     9: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup-config:23:in `<main>'
     8: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup-config:23:in `load'
     7: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup-config:109:in `<top (required)>'
     6: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
     5: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `load_sources'
     4: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `synchronize'
     3: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block in load_sources'
     2: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `map'
     1: from /home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block (2 levels) in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:20:in `id': wrong id called on {"uri"=>"maildir:/home/krerkkiat/personal", "usual"=>true, "archived"=>false, "id"=>1, "labels"=>[]} (RuntimeError)
IPv2 commented 4 years ago

Thank you @krerkkiat - that is extremely helpful.

I have been able to reproduce on my machine now, after upgrading my libyaml from 0.2.2 to 0.2.5

So I agree, yes, this is directly linked to https://github.com/yaml/libyaml/issues/196#issuecomment-644341507 that you found, and has started to fail because of the YAML spec change (the comma in !supmua.org,2006-10-01/Redwood/Maildir was valid in 1.1 but not 1.2, and libyaml 0.2.5 only supports YAML 1.2 with no backwards compatibility for 1.1).

If you want a quick fix to get up and running, you could manually change your sources.yaml and your installed Sup to use a non-standard modified YAML source, without the comma. For example:

# ~/.sup/sources.yaml
- !supmua.org/2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []
# ~/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb - line 29
# OLD: yaml_tag "!#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"
yaml_tag "!#{Redwood::YAML_DOMAIN}/#{Redwood::YAML_DATE}/#{path}"

But we'll need to workaround this in Sup itself, and this might cause you a headache down the line, because you may need to undo your sources.yaml change later when we release a new Sup version with a workaround.

I'll have a bit of a think on this one about how to tackle this. I suspect we'll need to introduce code in Sup to migrate from the existing YAML tag to a new YAML tag (without a comma).

IPv2 commented 4 years ago

I had a better idea for a different quick-fix workaround, which will get you up-and-running for now, and without modifying your configuration in ways that may cause headaches in future (for when later have a permanent workaround in Sup itself).

The idea is: download your own copy of an older version of libyaml (perhaps 0.2.4), and set the LD_LIBRARY_PATH environment variable when launching Sup, so that Sup alone uses it. You don't need to install the older libyaml system-wide, so no other programs would be affected.

Steps would be something like:

mkdir ~/libyaml
cd ~/libyaml
curl -o libyaml-dist-0.2.4.tar.gz https://codeload.github.com/yaml/libyaml/tar.gz/dist-0.2.4
tar -zxvf libyaml-dist-0.2.4.tar.gz
cd libyaml-dist-0.2.4/
./configure
make

# Then launch sup with an appropriate LD_LIBRARY_PATH
LD_LIBRARY_PATH=~/libyaml/libyaml-dist-0.2.4/src/.libs:$LD_LIBRARY_PATH sup

Note that you'll need to use a standard sources.yaml file, similar to the one in your first comment (not your later comment). This should work:

# sources.yaml
---
- !supmua.org,2006-10-01/Redwood/Maildir
  uri: maildir:/home/krerkkiat/mail/personal
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []
IPv2 commented 4 years ago

Alternative quick fix (per YAML 1.2 spec, section 6.8.2):

krerkkiat commented 4 years ago

Thank you for the help!

I went with the source code modifying method (the one at line lib/sup.rb:29), and I think I should be all good now in terms of the sources.yaml file.

I think I did try the last method already (using !<!supmua.org,2006-10-01/Redwood/Maildir>) unless I miss-understanding. Okay. I tried it again, and this is the result:

--- RuntimeError from thread: main
wrong id called on {"uri"=>"maildir:/home/krerkkiat/mail/personal", "usual"=>true, "archived"=>false, "sync_back"=>true, "id"=>1, "labels"=>["gmail"]}
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup.rb:20:in `id'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block (2 levels) in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `map'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:229:in `block in load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `synchronize'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/source.rb:228:in `load_sources'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/index.rb:102:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/lib/sup/util.rb:605:in `method_missing'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:146:in `<module:Redwood>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/gems/sup-1.0/bin/sup:76:in `<top (required)>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `load'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/sup:23:in `<main>'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
/home/krerkkiat/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
IPv2 commented 4 years ago

Thanks @krekkiat!

The "wrong id" error occurs when there's a mismatch between the tag expected by Sup in lib/sup.rb, and the tag in sources.yaml.

If you revert your manual changes to lib/sup.rb first (or reinstall the sup gem), and then make the sources.yaml change, then you should avoid that error.

(This is also why you might prefer to go down this route - because otherwise when you install a new Sup, you are likely to see that error message, if you use a non-standard tag.)

danc86 commented 3 years ago

I don't think there is anything wrong with using the comma character in tag URIs. The issue seems to be just with the syntax that sup was using in sources.yaml. Since upgrading to libyaml 0.2.5 I updated my sources.yaml to look like this, which I think is using the YAML tag support in the way that it was intended:

%TAG ! tag:supmua.org,2006-10-01/Redwood/
---
- !Maildir
  uri: ...

I also needed a corresponding fix in sup, to make it declare its actual complete tag URI (using tag:... instead of the ! syntax from YAML):

diff --git a/lib/sup.rb b/lib/sup.rb
index e90c971d..209ddea9 100644
--- a/lib/sup.rb
+++ b/lib/sup.rb
@@ -26,7 +26,7 @@ class Module
     props = props.map { |p| p.to_s }

     path = name.gsub(/::/, "/")
-    yaml_tag "!#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"
+    yaml_tag "tag:#{Redwood::YAML_DOMAIN},#{Redwood::YAML_DATE}/#{path}"

     define_method :init_with do |coder|
       initialize(*coder.map.values_at(*props))

What I haven't had time to figure out yet is, whether that patch will break on older versions of libyaml, and also how we can gracefully and safely migrate existing sources.yaml files with the previous incorrect syntax.

danc86 commented 2 years ago

I used podman run -it centos:7 to easily get an old Ruby environment. It has Psych 2.0.0 and libyaml 0.1.4. I confirmed that the change to declare the proper tag URI I wrote above works fine with these old versions.

It causes sup-add to emit sources.yaml with syntax like this, which appears to be the proper way of declaring a "global tag" and was probably what sup always intended:

---
- !<tag:supmua.org,2006-10-01/Redwood/Maildir>
  uri: maildir:///some/path
  usual: true
  archived: false
  sync_back: true
  id: 1
  labels: []

That also parses fine and works as intended on later libyaml.

The only remaining issue is how we can automatically fix up older sources.yaml when sup encounters them.