search

hitachi-nlp / appjsonify

A handy PDF-to-JSON conversion tool for academic papers implemented in Python.
MIT License
50 stars 3 forks source link

readme

appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit

appjsonify[^1] is a handy PDF-to-JSON conversion tool for academic papers implemented in Python. appjsonify allows you to obtain a structured JSON file that can be easily used for various downstream tasks such as paper recommendation, information extraction, and information retrieval from papers.
[^1]: Academic Paper PDF jsonify

appjsonify overview

Requirements

Installation

Prerequisites

If your environment does not have poppler, please install it. This is necessary to obtain PDF images using pdf2image. For more details, refer to Prerequisites.

Released version

pip install appjsonify
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

Editable version (Beta)

git clone https://github.com/hitachi-nlp/appjsonify.git
python -m pip install --editable .
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

Usage

appjsonify offers two options to structure your paper PDF file into a JSON file.

  1. Use the existing templates
    Suitable if a paper adopts the AAAI, ACL, ICML, ICLR, NeurIPS, IEEE, ACM, or Springer styles. See Templates for more details.
  2. Configure pipelines and parameters by yourself
    If a paper does not adopts the above formats, you need to specify the processing pipeline and its parameters. Please refer to Build your own pipeline for further information.

Templates

appjsonify provides two types of the templates for each of the following paper types: AAAI, ACL, ICML, ICLR, NeurIPS, IEEE, ACM, and Springer. One is more accurate but slower due to the use of machine learning based models, the other is less accurate but faster due to its rule based approach.

AAAI papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI

If your environment has a GPU(s), it is better to also specify --detectron_device_mode cuda to speed up the process.

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI2

ACL papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL2

ICML papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML2

ICLR papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR2

NeurIPS papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS2

IEEE papers

Currently only tested with IEEE BigData papers.

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE2

ACM papers

Currently only tested with TALLIP papers.

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM2

Springer papers

Better performance but slower

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer

Faster but a bit noisy

appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer2

Useful parameters

Build your own pipeline

appjsonify also allows users to build their own academic paper PDF-to-JSON processing pipeline. For more details, please refer to Available Modules and Document Handling in appjsonify.

How to add your own module

Users can add their own modules to appjsonify for more flexible document processing. To add modules, appjsonify must be installed in editable mode. See Customize appjsonify for more details and feel free to make a PR if you wish to add your module to this repository and package!

Contributing and Future Work

Contributions are more than welcome! Feel free to raise an issue and/or make a PR. Possible future work is as follows:

Citation

If you use appjsonify in your work, please cite the following.

@article{yamaguchi2023appjsonify,
      title={appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit}, 
      author={Atsuki Yamaguchi and Terufumi Morishita},
      year={2023},
      eprint={2310.01206},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

© 2023 Atsuki Yamaguchi and Terufumi Morishita (Hitachi, Ltd.)

This work is licensed under the MIT license unless specified.

appjsonify uses the follwoing publicly available works.