search

facebookresearch / nougat

Implementation of Nougat Neural Optical Understanding for Academic Documents
https://facebookresearch.github.io/nougat/
MIT License
8.98k stars 567 forks source link

Provide a end to end fine tune notebook. #56

Open imrankh46 opened 1 year ago

imrankh46 commented 1 year ago

Hi, I want to experiment, and fine tune nougat on other language. Please provide a notebook for fine tuning.

marwinsteiner commented 1 year ago

Check the readme, there are some instructions on how to fine-tune. You can just make the Jupyter notebook yourself...?

imrankh46 commented 1 year ago

Check the readme, there are some instructions on how to fine-tune. You can just make the Jupyter notebook yourself...?

Could you provide me with a sample dataset?

marwinsteiner commented 1 year ago

No, you have to make it yourself according to the instructions in the readme.

Also, related to fine-tuning: https://github.com/facebookresearch/nougat/issues/52#issuecomment-1713945716

imrankh46 commented 1 year ago

No, you have to make it yourself according to the instructions in the readme.

Also, related to fine-tuning: #52 (comment)

The readme are not helpful for me. That's why I am asking the question here. Should I need only pdf data and they will care to create data set ?

marwinsteiner commented 1 year ago

I am working on a little dataset generator for my own Nougat fine-tuning project. You have to create your own dataset for fine-tuning. I don't think they plan on generating a dataset for you... would be nice though ;). Could spin up a light streamlit app where you can drop all your PDFs and then dataset generation for Nougat fine-tuning would be handled under the hood... Then get your dataset and off you go to fine-tuning.

I would actually appreciate some more detail in the dataset generation section of the readme. I want to know why each step is being done and what is happening, not just step-by-step, and I guess @imrankh46, you would appreciate this as well.

Eventually Lukas or one of the other collaborators might give an update concerning fine-tuning so that might be worth waiting for, especially regarding https://github.com/facebookresearch/nougat/issues/52#issuecomment-1713945716. Then we'll know more regarding feasibility and concrete how-to steps I would hope.

It also always helps to read the paper/have a copy on hand. PDF: https://arxiv.org/abs/2308.13418

nullhook commented 1 year ago

I don't think they plan on generating a dataset for you... would be nice though

I'm confused, aren't there helpers inside ./nougat/dataset/ that does this for you? This section also says how

NielsRogge commented 1 year ago

Hi,

See also #128 with some tutorial notebooks.