search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.59k stars 3.51k forks source link

[feature request] im2markup or im2latex implementation in T2T #871

Open bezigon opened 6 years ago

bezigon commented 6 years ago

I need a specific T2T example for the im2markup task by @harvardnlp (arXiv) or im2latex task by @untrix (arXiv).

Both has his own approach to solving the problem, it would be interesting to adopt both options.

BTW @harvardnlp also implemented in most universal OpenNMT variant.

untrix commented 6 years ago

Hi @bezigon, I am the author of im2latex and am trying to understand your request. Are you asking for a new T2T 'model' and 'problem' for the image-to-latex task? Also, can you please let us know what you need this for? I will be happy to implement it if the T2T folks (@rsepassi ) accept this request.

bezigon commented 6 years ago

Hi @untrix! I'm newbie in deep learning, and I admire your work! Specifically in preprocessing and statistical analysis of training data.

To be honest, I would like to create a competitor of Mathpix :-) But in general, I would like to get experience with translating images into structured text. Here are scopes of this algorithm:

It is very important to understand better the work of CNN + LSTM + Attention, play with hyperparameters to get a more lightweight model, with the ability to run on a mobile devices.

And I understand that I lack some knowledge, but no harm in trying.

bezigon commented 6 years ago

Why T2T? Because, I believe, T2T will become the supreme algorithm.

The Master algorithm

untrix commented 6 years ago

Hi @bezigon , It seems that you are intending to experiment with the image-to-latex model in order to learn and perhaps build an app. Sounds like a good idea. Feel free to contact me directly by email as needed.

Best