JavaScript compatibility

[ggrossetie]

Hi,

I'm working on the Asciidoctor Browser extension (which uses Asciidoctor.js) and I got the following question:

Hi, Does the live preview support asciidoctor-bibtex? My bibliographies don't show up, but perhaps I need to configure something? Gavin

As far as I understand, "asciidoctor-bibtex" uses https://github.com/inukshuk/bibtex-ruby so it may not be an easy task to compile/port this library to JavaScript.

Anyway if you think it's possible to compile this library to JavaScript using Opal I can get you set-up :wink:

Thanks!

[ProgramFan]

asciidoctor-bibtex and bibtex-ruby are pure ruby and I can try to make asciidoctor-bibtex conform to a certain standard so that it can be compiled to JavaScript. asciidoctor-bibtex is such a simple package that the porting shall not be hard on its own. But you would have to check for all its dependencies.

[Nadano]

Would this allow to use asciidoc-bibtex in the live preview of vscode? that would be great!

[ggrossetie]

@Nadano Yes, the live preview of VSCode is based on Asciidoctor.js so once this library is transpiled to JavaScript it can be integrated in JavaScript-based tools.

[ProgramFan]

I am not familiar with opal at the moment. How can this be done?

[ggrossetie]

Basically, you will need to add a Rake task to produce a JavaScript file from this Ruby library. You can take a look at the Asciidoctor reveal.js converter:

https://github.com/asciidoctor/asciidoctor-reveal.js/blob/e443b549741782663d648ef40c37952624eafefa/Rakefile#L32-L50

require 'opal'

builder = Opal::Builder.new(compiler_options: {
  dynamic_require_severity: :error,
})
builder.append_paths 'lib'
builder.build 'asciidoctor-bibtex'

File.write 'asciidoctor-bibtex.js', builder.to_s

That's the easy part! Then you will probably need to make some adjustments to the code base to workaround some limitations/constraints dictated by Opal.

If you open a pull request with this Rake task, I can provide guidance to resolve these issues one step at a time (you will also need to run this task on Travis so we have a shared environment to reproduce issues).

[Leon0402]

Is there some progress on this topic?

I would help, in fact I even tried it a little bit myself, but I neither know ruby nor javascript very well making this quite difficult.

@Mogztter For example how does this work with dependencies. For instance asciidoctor.js, I tried specifying it like this: builder.use_gem 'asciidoctor'

It then stuck here: https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor.rb#L8

I then just disabled the require for asciidoctor completly as I saw that somewhere else require 'asciidoctor' unless RUBY_PLATFORM == 'opal'. But I'm unsure if this is the right way.

Similarly I have problems with other libs. I just did again something like builder.use_gem 'bibtex-ruby'. I think this will basically convert these libs to javascript as well?

Currently it is stuck with can't find file: "digest/md5" in:. Which seems to be a dependency of bibtex-ruby. Do I need to specify with "use_gem" all libs that are used as dependencies recursively?

[ggrossetie]

Similarly I have problems with other libs. I just did again something like builder.use_gem 'bibtex-ruby'. I think this will basically convert these libs to javascript as well?

Yes, you could say that.

I then just disabled the require for asciidoctor completly as I saw that somewhere else require 'asciidoctor' unless RUBY_PLATFORM == 'opal'. But I'm unsure if this is the right way.

This is the right way, Asciidoctor is a runtime dependency, you should not need it to transpile to JavaScript.

Currently it is stuck with can't find file: "digest/md5" in:. Which seems to be a dependency of bibtex-ruby. Do I need to specify with "use_gem" all libs that are used as dependencies recursively?

No, digest/md5 is part of the stdlib of Ruby: https://ruby-doc.org/stdlib-2.4.0/libdoc/digest/rdoc/Digest/MD5.html Unfortunately, Opal does not provide an implementation: https://github.com/opal/opal/issues/1661

Having said that, I didn't find any usage of Digest::MD5 in bibtex-ruby. Maybe this require can be removed? You should probably ask the maintainer.

[Leon0402]

This is the right way, Asciidoctor is a runtime dependency, you should not need it to transpile to JavaScript.

How do I know that it's only a runtime issue?

Having said that, I didn't find any usage of Digest::MD5 in bibtex-ruby. Maybe this require can be removed? You should probably ask the maintainer.

You're right, I did search a little bit with git blame, the code using it was removed already. I forked the project for now and removed it.

Next I run into an issue with dynamic require, but I think I solved this:

https://github.com/Leon0402/bibtex-ruby/commit/bcbdf04ebbd0e93c70e649381a156c528ecb067f

The next issue is that it requires "java".

A file required by "bibtex/filters.rb" wasn't found.
A file required by "bibtex/filters/latex.rb" wasn't found.
A file required by "latex/decode.rb" wasn't found.
A file required by "latex/decode/compatibility.rb" wasn't found.
can't find file: "java" in:

I don't quite understand it as I thought RUBY_PLATTFORM is "opal". Anyway I just completly excluded compatibility.rb the same way I did with asciidoctor for now (in my local gem). Which is probably a very bad idea. Do you know how to fix this properly

The next issue then is this line: https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/names.rb#L252

Based on your previous answer this seems again to be part of the stdlib and again not to be supported by opal: https://github.com/opal/opal/issues/1749? But I guess this code is important for the lib. So what can we do about it?

[ggrossetie]

I don't quite understand it as I thought RUBY_PLATFORM is "opal".

RUBY_ENGINE is equal to "opal", not sure about RUBY_PLATFORM.

Anyway I just completly excluded compatibility.rb the same way I did with asciidoctor for now (in my local gem). Which is probably a very bad idea. Do you know how to fix this properly

You can simplify the code for Opal by removing the if condition that contains the require 'java'. You can also add a special condition for Opal: if RUBY_ENGINE == 'opal'.

Based on your previous answer this seems again to be part of the stdlib and again not to be supported by opal: opal/opal#1749? But I guess this code is important for the lib. So what can we do about it?

That's correct but I don't think we are using to_xml in this library so we could probably remove it (when transpiling).

[Leon0402]

Do I then need to fork all these things and modify the source code? Or is there a better / easier way? (At least for the latex gem for instance ... I don't want to fork everything :-)) And just modifying it locally is not soo great.

RUBY_ENGINE is equal to "opal", not sure about RUBY_PLATFORM.

At least require 'asciidoctor' unless RUBY_PLATFORM == 'opal' worked fine, so I assumed it has to be opal

Edit: I see the issue. Opal does try to compile the whole file no matter what the variables are I think.

[ggrossetie]

At least require 'asciidoctor' unless RUBY_PLATFORM == 'opal' worked fine, so I assumed it has to be opal

You're right, both are defined: https://github.com/opal/opal/blob/master/opal/corelib/constants.rb#L1-L2

Edit: I see the issue. Opal does try to compile the whole file no matter what the variables are I think.

It only works when you explicitly check if the engine/platform is (or isn't) opal, see: https://github.com/opal/opal/blob/master/lib/opal/rewriters/opal_engine_check.rb

Do I then need to fork all these things and modify the source code? Or is there a better / easier way? (At least for the latex gem for instance ... I don't want to fork everything :-)) And just modifying it locally is not soo great.

I think the best approach right now is probably to modify the source code locally. Once you get something working, we will need to apply these changes before transpiling probably using monkey patch. That's what we are doing in Asciidoctor.js: https://github.com/asciidoctor/asciidoctor.js/blob/master/packages/core/lib/asciidoctor/js/asciidoctor_ext/document.rb. In this case, we replace the Document#fill_datetime_attributes method with a "JavaScript compatible" implementation.

If bibtex-ruby and/or latex-decode projects are willing to support Opal that's even better. In this case, you can submit patches upstream.

[Leon0402]

Okay I got something compiled yay :)

asciidoctor-bibtex.zip

What I have done up to now:

• Removed MD5 require from bibtex-ruby (fork)
• Removed all rdf and xml converter stuff from bibtex-rube (fork)
• Excluded compatibilities file of latex-decode, cls and citeproc (local)
• Remove all require and code of nokogiri from cls (local) -> Could be bad?
• And somehwere I removed some dynamic import, which might be bad as well (But I don't know where currently)

So I now try to use the extension:

npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc

But instantly get the error message from the js "ERROR: TypeError: $$ is not a function". This looks like jquery if I'm not mistaken.

[ggrossetie]

Okay I got something compiled yay :)

Yay! That's a first step, well done 👍🏻

Removed MD5 require from bibtex-ruby (fork)

I think you can submit this patch upstream since it's not used anymore. All other changes would probably be added via monkey-patch (at least at first).

But instantly get the error message from the js "ERROR: TypeError: $$ is not a function". This looks like jquery if I'm not mistaken.

Could you please copy/paste the complete stacktrace? JQuery is indeed using $ but Opal is also using $$ as an alias for Opal.const_get_relative: https://github.com/opal/opal/blob/584d8e758fa12b378b44e5beacc15e0298d51af9/opal/corelib/runtime.js#L420

You should use Opal 0.11.0 to transpile this library (because Opal 1.1.1 and Opal 0.11.0 are not compatible and Asciidoctor.js 2.2.x is using Opal 0.11.0). You can use the following declaration to use the exact same version as Asciidoctor.js (safest option): https://github.com/asciidoctor/asciidoctor-reveal.js/blob/ba16105cedb80171a127d543215b53db2b3f01a4/Gemfile#L6

[Leon0402]

Thanks! Using Opal 0.11.0 already helped with that.

Next issue with full stack trace:

ERROR:  datetime_format=: undefined method `datetime_format=' for #<Logger:0x476>
    at constructor.$method_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3938:56)
    at constructor.method_missing_stub (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1310:35)
    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9618:5
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9625:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.asciidoctor-bibtex/processor (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:21870:8)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17) {
  args: [ '%Y-%m-%d %H:%M:%S' ]
}

in the bibtext module section of the lib the following line is: Opal.add_stubs(['$require', '$new', '$key?', '$level=', '$-', '$datetime_format=', '$attr_accessor']);

Seems to be https://github.com/inukshuk/bibtex-ruby/blob/a2b74add5c5597cc193e307478193d1b566a9237/lib/bibtex.rb#L45

Removed it and next error. Is this how you deal with these issues? :D

The next error I could resolve myself. Now I get this:

ERROR:  SyntaxError: Invalid regular expression: /\\(r|H|u|v|G|M|d|c|k|b|B|t)\{(\p{L}\p{M}*)([[:alpha:]]*)\}/: Lone quantifier brackets
    at new RegExp (<anonymous>)
    at $Diacritics$4 (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3990:16)
    at Object.Opal.yieldX (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1475:18)
    at Function.$ruby_19 (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3758:19)
    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3987:233
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3998:9
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3999:7
    at Opal.modules.latex/decode/diacritics (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:4000:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)

The module is latex/decode/diacritics

      @patterns = [
        ruby_18 {
          /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{([[:alpha:]]?)([[:alpha:]]*)\}/ou          
        } ||
        ruby_19 {
          /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{(\p{L}\p{M}*)([[:alpha:]]*)\}/ou
        },
        /\\(o|O|l|L|aa|AA|ae|AE)\b/,
      ].freeze

[ggrossetie]

Removed it and next error. Is this how you deal with these issues? :D

Not really 😆 In this case, it's not crucial but the right way to fix it is to implement the missing method in Opal. Here, you should open a pull request to implement the method def datetime_format= in https://github.com/opal/opal/blob/master/stdlib/logger.rb

The next error I could resolve myself. Now I get this:

Ruby regular expression and JavaScript regular expression share some similarities but do not use the same engine and do not support the same features/syntaxes.

This regular expression is probably valid in Ruby but invalid in JavaScript. That's the reason why we are using the following block in Asciidoctor Ruby: https://github.com/asciidoctor/asciidoctor/blob/7d754d2ceb609ba54e83a058347ce64d3d843bec/lib/asciidoctor.rb#L427-L437 And in Asciidoctor.js, we override these constants: https://github.com/asciidoctor/asciidoctor.js/blob/master/packages/core/lib/asciidoctor/js/rx.rb

Having said that running /\\(r|H|u|v|G|M|d|c|k|b|B|t)\{(\p{L}\p{M}*)([[:alpha:]]*)\}/.test('foo') on the DevTools console (Chrome) is working fine:

Where do you get this error? Are you using Node? Which version?

[Leon0402]

Not really laughing In this case, it's not crucial but the right way to fix it is to implement the missing method in Opal. Here, you should open a pull request to implement the method def datetime_format= in https://github.com/opal/opal/blob/master/stdlib/logger.rb

Ah I see

Where do you get this error? Are you using Node? Which version?

It's from latex/decode/diacritics, the code snippet above or what do you mean?

The exact code should be:

new RegExp("" + "\\\\(" + ($send(self.macros.$keys(), 'map', [], ($$5 = function (k) {
  var self = $$5.$$s || this;
  if (k == null) {
    k = nil;
  };
  return $$($nesting, 'Regexp').$escape(k);
}, $$5.$$s = self, $$5.$$arity = 1, $$5)).$join("|")) + ")\\{(\\p{L}\\p{M}*)([[:alpha:]]*)\\}", 'u')

The regex should be more than what was outputed. At least the two "()" are missing and possibly the rest in the middle

I'm not sure I'm using Node? I guess :-) My Version is: v12.22.3

[ggrossetie]

It's from latex/decode/diacritics, the code snippet above or what do you mean?

Sorry I wasn't clear. What command do you run? npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc?

I'm not sure I'm using Node? I guess :-) My Version is: v12.22.3

If you are using asciidoctor-web-pdf then you are using Node.js. You can type node -v to get the version.

Could you please share the extension transpiled to JavaScript so I can troubleshoot/debug on my machine?

[ggrossetie]

I can reproduce this issue, [[:alpha:]] is not valid in a JavaScript regular expression. You need to replace it by \p{Alpha}.

@patterns = [
  ruby_19 {
    /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{(\p{L}\p{M}*)(\p{Alpha}*)\}/ou
  },
  /\\(o|O|l|L|aa|AA|ae|AE)\b/,
].freeze

(I removed ruby_18 because we don't need it when running with Opal)

[Leon0402]

npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc

Correct

If you are using asciidoctor-web-pdf then you are using Node.js. You can type node -v to get the version.

Yeah I do, it's v12.22.3

I can reproduce this issue, [[:alpha:]] is not valid in a JavaScript regular expression. You need to replace it by \p{Alpha}.

Why does it run in the browser developer console though? Is this Node specific?

ERROR:  constructor [ArgumentError]: wrong number of arguments (3 for 0)

  NOTE:If you want to enable passing a String argument please add "require 'opal-parser'" to your script

    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at Function.$exception (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:4993:14)
    at Function.$raise (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:4681:31)
    at Function.$module_eval (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3320:32)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5255:12
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5304:7
    at Opal.modules.bibtex/name_parser (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5305:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9681:8)

Her the js asciidoctor-bibtex.zip

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

[ggrossetie]

Why does it run in the browser developer console though? Is this Node specific?

I didn't understand 🤔

Her the js asciidoctor-bibtex.zip

Thanks!

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

That's bad 😞 But yes, since bibtex is evaluating Ruby as runtime, we need the opal-parser library to convert Ruby code as JavaScript code at runtime. To be honest, I've never used this library so I can't really help you but I'm pretty sure it will make things complicated since we will need to read a file named names.y at runtime.

I'm not entirely sure why this is not done at compile time (in bibtex) since module_eval can be resolve statically (at least in https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb)

[Leon0402]

Why does it run in the browser developer console though? Is this Node specific?

I didn't understand thinking

Her the js asciidoctor-bibtex.zip

Thanks!

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

That's bad disappointed But yes, since bibtex is evaluating Ruby as runtime, we need the opal-parser library to convert Ruby code as JavaScript code at runtime. To be honest, I've never used this library so I can't really help you but I'm pretty sure it will make things complicated since we will need to read a file named names.y at runtime.

I'm not entirely sure why this is not done at compile time (in bibtex) since module_eval can be resolve statically (at least in https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb)

Perhaps because of:

#
# DO NOT MODIFY!!!!
# This file is automatically generated by Racc 1.4.15
# from Racc grammer file "".
#

See beginning of file.

As far as I can see it's name_parser.rb and parser.rb, which are definitly needed :(

Any suggestions how to move forward? This sounds like it might be simpler to rewrite this whole thing manually in javascript :disappointed:

[ggrossetie]

I'm not familiar with racc but apparently it's possible to regenerate files using the -l. In this case, module_eval is not used:

lib/bibtex/name_parser.rb

Racc_debug_parser = false

##### State transition tables end #####

# reduce 0 omitted

def _reduce_1(val, _values, result)
 result = []
    result
end

# reduce 2 omitted

def _reduce_3(val, _values, result)
 result = [val[0]]
    result
end

def _reduce_4(val, _values, result)
 result << val[1]
    result
end

def _reduce_5(val, _values, result)
 result = Name.new(val[1])
    result
end

I think you should suggest this change at: https://github.com/inukshuk/bibtex-ruby

[Leon0402]

Wow awesome! :+1:

I regenerated the files in my fork: https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb

The error message hasn't changed though.

Here is a snippet from the js code. self.$module_eval("" + "\n" + " @patterns = {\n" + " :and => /,?\\s+and\\s+/io,\n" + " :space => /\\s+/o,\n" + " :comma => /,/o,\n" + " :lower => /[[:lower:]][[:lower:][:upper:]]*/uo,\n" + " :upper => /[[:upper:]][[:lower:][:upper:].]*/uo,\n" + " :other => /[^\\s,\\{\\}\\[:upper:][:lower:]]+/uo,\n" + " :lbrace => /\\{/o,\n" + " :rbrace => /\\}/o,\n" + " :braces => /[\\{\\}]/o,\n" + " :escape => /\\./o,\n" + " :last_first => /[\\p{Han}\\p{Hiragana}\\p{Katakana}\\p{Hangul}]/uo\n" + " }\n" + "\n" + " class << self\n" + " attr_reader :patterns\n" + " end\n" + "\n" + " def initialize(options = {})\n" + " self.options.merge!(options)\n" + " end\n" + "\n" + " def options\n" + " @options ||= { :debug => ENV['DEBUG'] == true }\n" + " end\n" + "\n" + " def parse(input)\n" + " @yydebug = options[:debug]\n" + " scan(input)\n" + " do_parse\n" + " end\n" + "\n" + " def next_token\n" + " @stack.shift\n" + " end\n" + "\n" + " def on_error(tid, val, vstack)\n" + " BibTeX.log.debug(\"Failed to parse BibTeX Name on value %s (%s) %s\" % [val.inspect, token_to_str(tid) || '?', vstack.inspect])\n" + " end\n" + "\n" + " def scan(input)\n" + " @src = StringScanner.new(input)\n" + " @brace_level = 0\n" + " @last_and = 0\n" + " @stack = [[:ANDFL,'(^start)']]\n" + " @word = [:PWORD,'']\n" + " do_scan\n" + " end\n" + "\n" + " private\n" + "\n" + " def do_scan\n" + " until @src.eos?\n" + " case\n" + " when @src.scan(NameParser.patterns[:and])\n" + " push_word\n" + " @last_and = @stack.length\n" + " @stack.push([:ANDFL,@src.matched])\n" + "\n" + " when @src.skip(NameParser.patterns[:space])\n" + " push_word\n" + "\n" + " when @src.scan(NameParser.patterns[:comma])\n" + " push_word\n" + " @stack.push([:COMMA,@src.matched])\n" + "\n" + " when @src.scan(NameParser.patterns[:lower])\n" + " is_lowercase\n" + " @word[1] << @src.matched\n" + "\n" + " when @src.scan(NameParser.patterns[:upper])\n" + " is_uppercase\n" + " @word[1] << @src.matched\n" + "\n" + " when @src.scan(NameParser.patterns[:other])\n"

And a couple more of "module_eval" all related to these two files.

There is only one "eval" left https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/filters.rb#L10. But this doesn't seem at all related?

[ggrossetie]

There's still a self.$module_eval in the JavaScript file but I have no idea why... 🤔

And a couple more of "module_eval" all related to these two files.

That's weird... I can't find any module_eval in the source (Ruby) so I'm not sure why Opal generates self.$module_eval.

But this doesn't seem at all related?

I don't think it's related, here we are using class_eval.

[Leon0402]

I figured it out. First there was a problem with another lib we have as an indirect dependency https://github.com/berkmancenter/namae. This seems pretty much the same code generated with racc, so I fixed it with the -l flag.

And then it didn't use my newest master of the bibtex fork. And I still don't understand why. I put branch "master" in my gem file and it worked all the time. I now pinned it to the newest ref on master and now it's installing that version with "bundle install".

No module_eval anymore :-)

The next error is regex again:

Invalid regular expression: /[[:lower:]][[:lower:][:upper:]]*/: Lone quantifier brackets

These come from names.y in ruby-bibtex.

  @patterns = {
    :and => /,?\s+and\s+/io,
    :space => /\s+/o,
    :comma => /,/o,
    :lower => /[[:lower:]][[:lower:][:upper:]]*/uo,
    :upper => /[[:upper:]][[:lower:][:upper:].]*/uo,
    :other => /[^\s,\{\}\\[:upper:][:lower:]]+/uo,
    :lbrace => /\{/o,
    :rbrace => /\}/o,
    :braces => /[\{\}]/o,
    :escape => /\\./o,
    :last_first => /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/uo
  }

I suppose it will be similar to the fix you had above with [[:xyz:] becoming to \p{xyz}. I'm not sure though how to replace [[:xyz:][:abc:]]

[Leon0402]

I'm note 100% sure, but I think I figured this out:

https://github.com/Leon0402/bibtex-ruby/commit/c605037343517011b1a87695ec133adeb609f390

It now says

SyntaxError: Invalid regular expression: /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/: Invalid property name in character class

I think the /u in the end might be the problem?

"braces": /[\{\}]/, "escape": /\\./, "last_first": /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/u});

at least without there is no problem. Can I just leave it away in ruby?

ERROR:  Iconv: uninitialized constant BibTeX::Iconv
    at Function.$const_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3083:52)
    at const_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:224:32)
    at Opal.const_get_relative (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:299:38)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5717:20
    at Opal.modules.bibtex/compatibility (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5741:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:10355:8)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)

Which might come from https://github.com/inukshuk/bibtex-ruby/blob/a2b74add5c5597cc193e307478193d1b566a9237/lib/bibtex/compatibility.rb. But I don't see the problem.

[ggrossetie]

I think the reason is that the unicode property escapes Han, Hiragana, Katakana and Hangul are not supported in JavaScript.

at least without there is no problem. Can I just leave it away in ruby?

If you remove the /u the regular expression won't work the same. I'm not sure why the Bibtex library is matching Han, Hiragana, Katakana, and Hangul unicode characters? Please note that it's possible to replace unicode property escapes with unicode range. For instance, if we want to match Kanji, we can use: /[\u4E00-\u9FAF\u3040-\u3096\u30A1-\u30FA\uFF66-\uFF9D\u31F0-\u31FF]/.

See: https://stackoverflow.com/questions/7344871/javascript-regular-expression-to-catch-kanji

Which might come from inukshuk/bibtex-ruby@a2b74ad/lib/bibtex/compatibility.rb. But I don't see the problem.

Iconv is a wrapper class for the UNIX 95 iconv() function family. I doubt it will work in a JavaScript environment. You should update the code as follows:

module BibTeX
  @iconv_replacements = Hash['ä', 'ae', 'ö', 'oe', 'ü', 'ue', 'Ä', 'Ae', 'Ö', 'Oe', 'Ü', 'Ue', 'ß', 'ss']

  # Returns +str+ transliterated containing only ASCII characters.
  def self.transliterate(str)
    str.gsub(/[äöüÄÖÜß]/, @iconv_replacements)
  end
end

[Leon0402]

I'm not sure why the Bibtex library is matching Han, Hiragana, Katakana, and Hangul unicode characters?

I have no idea either, but I replaced it and it at least compiles. We will see if it works in the end :-)

You should update the code as follows:

Oh right, yeah! That's an easy fix, thanks.

ERROR:  []: super: no superclass method `[]' for {"date"=>["accessed", "container", "event-date", "issued", "original-date", "submitted"], "names"=>["author", "collection-editor", "composer", "container-author", "recipient", "editor", "editorial-director", "illustrator", "interviewer", "original-author", "translator", "director", "reviewed-author"], "number"=>["chapter-number", "collection-number", "edition", "issue", "number", "number-of-pages", "number-of-volumes", "volume"], "text"=>["abstract", "annote", "archive", "archive_location", "archive-place", "authority", "call-number", "citation-label", "citation-number", "collection-title", "container-title", "container-title-short", "dimensions", "DOI", "event", "event-place", "first-reference-note-number", "genre", "ISBN", "ISSN", "jurisdiction", "keyword", "locator", "medium", "note", "original-publisher", "original-publisher-place", "original-title", "page", "page-first", "PMID", "PMCID", "publisher", "publisher-place", "references", "reviewed-title", "scale", "section", "source", "status", "title", "title-short", "URL", "version", "year-suffix"]}
    at Object.Opal.find_super_dispatcher (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1386:32)

This seems to be https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/entry/citeproc_converter.rb.

Is something with Hash not supported?

[ggrossetie]

Your last error does not ring a bell. Probably related to this "advanced" syntax:

Hash.new { |_h, k| k }.merge(Hash[*%w[date issued].map(&:intern)]).freeze

Apparently, the star symbol is a "Variable Length Argument List, Asterisk Operator": https://stackoverflow.com/questions/4170037/what-does-the-star-mean-in-ruby 😆

This syntax is probably not supported by Opal. For reference, it produces:

{:date=>:issued}

I'm pretty sure we can use another syntax to get the same result.

[ggrossetie]

The following does the same:

CSL_FILTER = %w[
    date issued
    isbn ISBN
    booktitle container-title
    journal container-title
    journaltitle container-title
    series collection-title
    address publisher-place
    pages page
    number issue
    url URL
    doi DOI
    pmid PMID
    pmcid PMCID
    year issued
    type genre
    school publisher
    institution publisher
    organization publisher
    howpublished publisher
    type genre
    urldate accessed
  ]
  .map(&:intern)
  .each_slice(2)
  .to_a.reduce({}) { |memo, element| memo[element[0]] = element[1]; memo}

Maybe it compiles?

[ggrossetie]

The following is also working:

CSL_FILTER = Hash[%w[
    date issued
    isbn ISBN
    booktitle container-title
    journal container-title
    journaltitle container-title
    series collection-title
    address publisher-place
    pages page
    number issue
    url URL
    doi DOI
    pmid PMID
    pmcid PMCID
    year issued
    type genre
    school publisher
    institution publisher
    organization publisher
    howpublished publisher
    type genre
    urldate accessed
  ].map(&:intern).each_slice(2).to_a]

[Leon0402]

hmm, I think I was mistaken, I'm sorry. Based on some more looking in the source code and experiment it seems to have something to do with: https://github.com/inukshuk/citeproc/blob/master/lib/citeproc/variable.rb#L15

Although I'm not sure what line exactly it is or if it's even that file. Based on the error message and the code around that error message, it might be that: https://github.com/inukshuk/citeproc/blob/master/lib/citeproc/extensions.rb#L29

This seems to also somewhat match the error message, although I'm not yet sure how the connection with variable.rb is.

I will have another look later. Here is the current js file, if you wanna have a look too asciidoctor-bibtex_1.zip

[ggrossetie]

You can try to remove the following lines:

https://github.com/inukshuk/citeproc/blob/95af849fe640b0c73d0d8d4eaea62f1ce2cb2bc1/lib/citeproc/extensions.rb#L107-L108

I did some tests and DeepFetch#[] is never called...

[Leon0402]

Thanks, that works!

Our next problem is from csl-ruby. https://github.com/inukshuk/csl-ruby/blob/master/lib/csl/parser.rb#L11 and https://github.com/inukshuk/csl-ruby/blob/master/lib/csl/schema.rb. Originally I just commented out these lines, but I think I shouldn't do this, some parser is probably required to read the csl styles.

For using nokogiri, I probably need to add this to the dependencies. But that leads to errors with nokogiri. Otherwise we can use the fallback REXML, but that is not supported as you said earlier, right?

So I guess we need to take the nokogiri approach and compile that to javascript as well?

[mojavelinux]

So I guess we need to take the nokogiri approach and compile that to javascript as well?

There is no possible way you are going to be able to compile nokogiri to JavaScript. It uses native libraries (libxml2) and that's outside (like way outside) of what Opal can transpile.

[ggrossetie]

I wonder if it would be possible to create an abstraction on top of the citeproc Ruby library in order to replace the implementation by https://github.com/Juris-M/citeproc-js or https://github.com/larsgw/citation.js

[Leon0402]

Do you think it's worth it? I really wonder if it wouldn't be easier, based on our experience here, to write a native implementation on top of existing javascript libraries like the one you linked? I would be willing to try that as well with some help

[ggrossetie]

Do you think it's worth it? I really wonder if it wouldn't be easier, based on our experience here, to write a native implementation on top of existing javascript libraries like the one you linked?

Hard to say, I don't know how much logic we will need to reimplement. It seems that this extension does quite a few things. I think we should not underestimate the maintenance burden.

Having said that, I don't know if citeproc.js can fully substitue the citeproc and csl-styles Ruby libraries but I think it's worth a try. My idea is to create a CiteProc module in this extension to "hide" the citeproc and csl Ruby libraries. When transpiling to JavaScript, we will be able to replace this module (and thus, we won't need to transpile citeproc, csl-styles or any transitive Ruby dependencies).

The good thing is that evaluating citeproc.js will be useful even if we decide to write a native implementation so I think this is the right approach.

[Leon0402]

Having said that, I don't know if citeproc.js can fully substitue the citeproc and csl-styles Ruby libraries but I think it's worth a try. My idea is to create a CiteProc module in this extension to "hide" the citeproc and csl Ruby libraries. When transpiling to JavaScript, we will be able to replace this module (and thus, we won't need to transpile citeproc, csl-styles or any transitive Ruby dependencies).

Could you outline a little bit more how that replacement works in detail? We have a module CiteProc with some interface using the citeproc and csl libraries. Then we add require 'CiteProc' unless RUBY_PLATTFORM = 'Opal', so it's just ignored when transpiling. So the javascript libraries will have calls to the interface, but missing the interface itself. Then we write a javascript module with the exakt same methods, but a native implementation and add that to the javascript file? Is that about right?

Does Opal has some mechanism to bundle our own native javascript code? Or do we need to manually put them together?

[ggrossetie]

Could you outline a little bit more how that replacement works in detail? We have a module CiteProc with some interface using the citeproc and csl libraries.

That's correct.

Then we add require 'CiteProc' unless RUBY_PLATTFORM = 'Opal', so it's just ignored when transpiling. So the javascript libraries will have calls to the interface, but missing the interface itself. Then we write a javascript module with the exakt same methods, but a native implementation and add that to the javascript file? Is that about right?

Yes, I think we need to do something like:

if RUBY_ENGINE == 'opal'
  require 'asciidoctor-bibtex/js'
else
  require 'asciidoctor-bibtex/citeproc'
end

Then, we can provide a JavaScript implementation:

asciidoctor-bibtex/js.rb

require 'asciidoctor-bibtex/js/citeproc'

asciidoctor-bibtex/js/citeproc.rb

module AsciidoctorBibtex
  `const __cite__ = require('citation-js')`
  module CiteProc
    def format(citation, style) 
      %x{
        const data = new __cite__(citation)
        return data.format('bibliography', {
          format: 'html',
          template: style
        })
      }
    end
  end
end

Does Opal has some mechanism to bundle our own native javascript code? Or do we need to manually put them together?

Everything wrapped in %x{} is "raw" code and will be passed as-is by the compiler/transpiler to the output. You can also use backticks when you want to declare inline code. For instance:

value = `Math.floor(5.95)`

In the above example, value is a Ruby variable but the code between backticks is "raw" code. Does it make sense?

[Leon0402]

Does it make sense?

I think it does! I will give it a shot :)

[Leon0402]

@Mogztter I would like to give this another shot. So I redid the whole process and this time documented it a little bit better :-) I'm again at the step that I have an exported javascript file. But when I try to build a pdf with:

npm x asciidoctor-web-pdf --require ../asciidoctor-bibtex/dist/asciidoctor-bibtex.js main.adoc

the extension doesn't get used.

I figured there might be a problem with registering the extension and had a look at revealjs. So I added such a file https://github.com/asciidoctor/asciidoctor-reveal.js/blob/master/src/asciidoctor-revealjs.tmpl.js#L12 and the corresponding code in rake. I'm not sure though what the main module mainModule = Opal.const_get_qualified(Opal.Asciidoctor, 'Revealjs') should be. I tried AsciidoctorBibtex because I found such lines in the jsvar self = $module($base, 'AsciidoctorBibtex');. But this didn't work.

Thanks!

[Leon0402]

Okay I think I actually was not on the right track with the template. I tried it without it and using asciidoctorjs as suggested by @mojavelinux in zulip. This allows me to actually use the extension for now.

I got a bunch of errors regarding some Opal methods that are not found like Opal.queue or Opal.return_val. It seems they were introduced in newer version of opal, which asciidoctor apparently does not support? So I switched back to the one here: https://github.com/asciidoctor/asciidoctor-reveal.js/blob/master/Gemfile

Isn't it possible to use newer versions of Opal? There were quite some releases since then that might make our life easier

[mojavelinux]

You can't use newer versions of Opal than what Asciidoctor.js is tested with. In a sense, you need to think of Opal as being bundled with Asciidoctor.js. They have a very tight relationship and any changes to Opal could break Asciidoctor.js.

[Leon0402]

Taking the discussion to a zulip topic: https://asciidoctor.zulipchat.com/#narrow/stream/279645-users.2Fasciidoctor.2Ejs/topic/Port.20Asciidoctor.20BibTex/near/279793429