search

kermitt2 / grobid

A machine learning software for extracting information from scholarly documents
https://grobid.readthedocs.io
Apache License 2.0
3.54k stars 452 forks source link

How to Enable GUP server(Ubuntu) feature on Grobid Training? #316

Open YangaPri opened 6 years ago

YangaPri commented 6 years ago

Hi,

Grobid training on new huge dataset took more days and months for training.

We are having Ubuntu GUP server, while executing Phyton related class files it was to faster in the training. But Grobid was developed under 'Java'.

Is there any way to enable GUP server feature on training? In order to make training of dataset to be faster?

Could anyone please help on type kind of issue.

Thank you in advance.

lfoppiano commented 6 years ago

Hi @YangaPri your issue is not really clear. What is Ubuntu GUP server?

I think usually Java is faster than python by design ;-)

kermitt2 commented 6 years ago

Hello @YangaPri !

Maybe you are talking about using GPU?

GROBID uses Wapiti as CRF engine which is written in C but does not support GPU for training. As far as I know, there is no equivalent Python related CRF engine.

Regarding GPU, I am experimenting Deep Learning models for GROBID in https://github.com/kermitt2/delft (DeLFT) which uses GPU and indeed is significantly much faster for training, but all the models are for the moment less accurate than current CRF models by 2-3 points in f-score.

Regarding your training time issue, for me all the training time using multi-threading are under one day even for the largest model on a 8-core machine. Are you doing semi-supervised training for having such huge training dataset ?