Chinese charset when convent adoc into pdf file

[Walter0507]

Dear: I'm a developer from China. When I try to convent a adoc file into pdf, I met a strange issue. It didn't convent all Chinese words in pdf file, but just some of them.

Here below is my code:

        // Then validate that three AsciiDoc files have been created
        Asciidoctor act = Factory.create();
        File[] files = new File(path).listFiles();
        for (File file : files) {
            act.renderFile(file, OptionsBuilder.options()
                    .backend("pdf").get());
//          act.renderFile(file, new HashMap<String, Object>());
        }

The maven configured like this:

        <dependency>
            <groupId>org.asciidoctor</groupId>
            <artifactId>asciidoctorj</artifactId>
            <version>1.5.2</version>
        </dependency>
    <dependency>
        <groupId>org.asciidoctor</groupId>
        <artifactId>asciidoctorj-pdf</artifactId>
        <version>1.5.0-alpha.9</version>
    </dependency>

Have you met this issue before? I also checked the issue list in github, but no one faced this issue. Could you give some help about this?

[mojavelinux]

@chloerei is working on this. Please see https://github.com/chloerei/asciidoctor-pdf-cjk. It's just a matter of providing the fonts. PDF is "bring your own fonts". CJK fonts tend to be very large and make the gem file too big for an international user base. Therefore, we've decided to make a dedicated gem to provide the necessary theme and fonts. I'm sure Rei would appreciate your feedback.

I recommend trying to use it with Ruby first, then following the instruction for loading a custom gem file when running on AsciidoctorJ. (see https://github.com/asciidoctor/asciidoctorj#loading-ruby-libraries).

[mojavelinux]

...it's also important to point out that CJK languages really need their own theme anyway because there are other changes necessary (such as justification and font size) that these languages need in order to render nicely. So I think a dedicated gem and theme is the way to go.

[mojavelinux]

Related to https://github.com/asciidoctor/asciidoctor-pdf/issues/82.

Please note that this is an Asciidoctor PDF issue, not an AsciidoctorJ issue.

[Walter0507]

@mojavelinux After reading the documents(not familiar with Ruby), I'm still confused.

I have checked the two projects, it seems chloerei already finished this.

I also have a look at the asciidoctorj-pdf.jar. It's written by Ruby. So, I should re-package this jar(add cjk feather) and replace the old one. Is that right?

[mojavelinux]

I pointed to the wrong repository. asciidoctorj-pdf-cjk solves a different problem (also important). The repository with the theme and fonts is this one:

https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic

You don't need to modify AsciidoctorJ PDF. But you are going to have to do a bit of work to get the gem on the classpath and loaded. This is an area that simply needs to be documented more clearly. So you are working on the cutting edge.

You need to install the gem as part of the build into a folder you can add to the classpath. This can be accomplished using either Maven or Gradle (there are plugins for both). Here's an example with Maven:

https://github.com/asciidoctor/asciidoctor-maven-examples/blob/0b50187cf334858936e15e0e90e7ff41537be766/asciidoctor-diagram-example/pom.xml

(replace asciidoctor-diagram with asciidoctor-pdf-cjk-kai_gen_gothic)

Then, you need to set the following attribute when calling the Asciidoctor API:

.attribute("pdf-style", "KaiGenGothicCN")

You also need to require the gem:

asciidoctor.requireLibrary("asciidoctor-pdf-cjk-kai_gen_gothic")

To help you move forward, it is probably best if you setup a test project, push it to GitHub and we can hack on it together. Then we can use that project as an example to show others how it is done.

[chloerei]

I want to notice asciidoctor-pdf-cjk-kai_gen_gothic need to download fonts from github releases after installed, I write a script in exe/asciidoctor-pdf-cjk-kai_gen_gothic-install to do this. I'm not familiar with Java.

[chloerei]

Because fonts set for all CJK charsets is large size, you can download only KaiGenGothicCN-* and RobotoMono-* fonts in https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/releases/tag/v0.1.0-fonts by hand.

[mojavelinux]

That's for the clarification @chloerei.

You might want to consider downloading these fonts as part of the gem install. You can see a similar strategy in the gem for kindlegen.

https://github.com/tdtds/kindlegen

That would save the user from having to do this step manually. I'm not sure whether this works when installing the gem from the Maven or Gradle plugin, though I suppose it could be added.

@Walter0507 you'd have to get Maven to download these fonts for now and put them into the data/fonts directory inside the extracted asciidoctor-pdf-cjk-kai_gen_gothic gem folder after it is installed (or just put them in the repository and get Maven to move them around).

[mojavelinux]

To clarify my point about the gem install, there is a post install hook that allows you to run a script. That's what you'd be tapping into.

[mojavelinux]

I'm not familiar with Java.

We can sort that part out.

[mojavelinux]

I got this working with Maven. Here's how I did it.

First, declare the repository where the gems can be fetched:

<repositories>
    <repository>
       <id>rubygems-proxy-releases</id>
        <url>http://rubygems-proxy.torquebox.org/releases</url>
    </repository>
</repositories>

Next, configure the gem-maven-plugin to fetch and install the gems into the build (aka target) directory:

<plugin>
    <groupId>de.saumya.mojo</groupId>
    <artifactId>gem-maven-plugin</artifactId>
    <version>1.1.3</version>
    <executions>
        <execution> 
            <goals><goal>sets</goal></goals>
        </execution>
    </executions>
    <configuration>
        <jrubyVersion>9.0.4.0</jrubyVersion>
        <gemHome>${project.build.directory}/gems</gemHome>
        <gemPath>${project.build.directory}/gems</gemPath>
        <scope>provided</scope>
        <gems>      
            <asciidoctor-pdf-cjk>0.1.2</asciidoctor-pdf-cjk>
            <asciidoctor-pdf-cjk-kai_gen_gothic>0.1.1</asciidoctor-pdf-cjk-kai_gen_gothic>
        </gems>     
    </configuration>
</plugin>

Next, download the fonts:

cd target/gems-provided/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1
ruby exe/asciidoctor-pdf-cjk-kai_gen_gothic-install
cd -

Finally, configure the Asciidoctor Maven plugin: (I realize you are using AsciidoctorJ as a library, not in a Maven build, but this should still show you what is needed).

<plugin>
    <groupId>org.asciidoctor</groupId>
    <artifactId>asciidoctor-maven-plugin</artifactId>
    <version>${asciidoctor.maven.plugin.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.jruby</groupId>
            <artifactId>jruby-complete</artifactId>
            <version>${jruby.version}</version>
        </dependency>
        <dependency>
            <groupId>org.asciidoctor</groupId>
            <artifactId>asciidoctorj</artifactId>
            <version>1.5.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.asciidoctor</groupId>
            <artifactId>asciidoctorj-pdf</artifactId>
            <version>1.5.0-alpha.10.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <gemPath>${project.build.directory}/gems-provided</gemPath>
        <requires>
            <require>asciidoctor-pdf-cjk-kai_gen_gothic</require>
        </requires>
    </configuration>
    <executions>
        <execution>
            <id>generate-pdf</id>
            <phase>generate-resources</phase>
            <goals>
                <goal>process-asciidoc</goal>
            </goals>
            <configuration>
                <backend>pdf</backend>
                <attributes>
                    <pdf-style>KaiGenGothicCN</pdf-style>
                </attributes>
            </configuration>
        </execution>
    </executions>
</plugin>

I hope that helps get you going.

[Walter0507]

I still failed for installing asciidoctor-pdf-cjk-kai_gen_gothic component. In fact, I have no idea how to install this component into my asciidoctorj-pdf.jar files.

Please find the jar file structure in attachments. I guess After installing, the gem package should have a sub folder named 'asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1', then we should put the KaiGenGothicCN into folder './data/fonts/'. This should be means we also support cjk component.

When calling asciidoctor api, we should include the cjk support like this: asciidoctor.rubyExtensionRegistry().requireLibrary("asciidoctor-pdf-cjk-kai_gen_gothic");

and also set attributes like: Map<String, Object> attributes = new HashMap<String, Object>(); attributes.put("pdf-style", "KaiGenGothicCN"); Attributes attr = new Attributes(); attr.setAttributes(attributes);

Then this should be ok.

But the problem is I still don't know how to create the new jar file(exactly adding the cjk component into the jar file.). Could you help to solve this issue?

I can download the file from url: https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/releases/tag/v0.1.0-fonts. So right now, all I need is add the component 'asciidoctor-pdf-cjk-kai_gen_gothic' into the gem folder in attachment

![Uploading asciidoctor-pdf_structure.jar.png…]()

[Walter0507]

I just take a look at 'asciidoctor-pdf' project, and also search a file: https://github.com/asciidoctor/asciidoctor-pdf/blob/master/asciidoctor-pdf.gemspec like this: s.add_development_dependency 'rake', '~> 10.0'

s.add_development_dependency 'rdoc', '~> 4.1.0'

s.add_runtime_dependency 'asciidoctor', '~> 1.5.0'

Prawn 2.x requires Ruby 2.x, so cast a wider net

s.add_runtime_dependency 'prawn', '>= 1.3.0', '< 3.0.0' s.add_runtime_dependency 'prawn-table', '0.2.2' s.add_runtime_dependency 'prawn-templates', '0.0.3' s.add_runtime_dependency 'prawn-svg', '0.21.0' s.add_runtime_dependency 'prawn-icon', '1.0.0' s.add_runtime_dependency 'safe_yaml', '~> 1.0.4' s.add_runtime_dependency 'thread_safe', '~> 0.3.5'

For our usage, treetop 1.6.2 is slower than 1.5.3

s.add_runtime_dependency 'treetop', '1.5.3' end

I think you can add one more line like this? s.add_development_dependency 'rake', '~> 10.0'

s.add_development_dependency 'rdoc', '~> 4.1.0'

s.add_runtime_dependency 'asciidoctor', '~> 1.5.0'

Prawn 2.x requires Ruby 2.x, so cast a wider net

s.add_runtime_dependency 'prawn', '>= 1.3.0', '< 3.0.0' s.add_runtime_dependency 'prawn-table', '0.2.2' s.add_runtime_dependency 'prawn-templates', '0.0.3' s.add_runtime_dependency 'prawn-svg', '0.21.0' s.add_runtime_dependency 'prawn-icon', '1.0.0' s.add_runtime_dependency 'safe_yaml', '~> 1.0.4' s.add_runtime_dependency 'thread_safe', '~> 0.3.5'

For our usage, treetop 1.6.2 is slower than 1.5.3

s.add_runtime_dependency 'treetop', '1.5.3' s.add_runtime_dependency 'asciidoctor-pdf-cjk-kai_gen_gothic', '0.1.1' end

And then package it into a jar. It should be ok?

[mojavelinux]

I have no idea how to install this component into my asciidoctorj-pdf.jar files.

You don't put anything in your asciidoctorj-pdf.jar file (unless that is the name of your application, in which case, read on...).

You need to put the entire contents of the target/gems-provided directory onto the classpath (and thus, inside a jar file). That's all you need to do.

Within that classpath, you need to make sure you have the following directory with the CJK fonts inside.

gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/data/fonts

To get it to that location, the fonts should first be put into this directory:

target/gems-provided/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/data/fonts

You first have to run the Maven build to get the gems installed into target/gems-provided. Then you need to download the fonts. Then you can package the jar.

---

Btw, if you are using AsciidoctorJ 1.5.3.2, you can use:

asciidoctor.requireLibrary("asciidoctor-pdf-cjk-kai_gen_gothic")

[mojavelinux]

You need to put the entire contents of the target/gems-provided directory onto the classpath (and thus, inside a jar file).

Technically, you only need these folders in your jar:

• target/gems-provided/specifications => specifications
• target/gems-provided/gems => gems

The other folders in target/gems-provided are only temporary files.

[mojavelinux]

I just take a look at 'asciidoctor-pdf' project

You don't need to be looking there.