search

uclnlp / jack

Jack the Reader
MIT License
257 stars 82 forks source link

Unify location of all NLP pre-processing #265

Closed dirkweissenborn closed 6 years ago

dirkweissenborn commented 6 years ago

@pminervini I will try to unify nlp preprocessing and clean up existing code now.

TimDettmers commented 6 years ago

Will your preprocessing style be affected? I am currently building on the FastQA preprocessing pipeline.

dirkweissenborn commented 6 years ago

@TimDettmers yes it will, however, only slightly so it should be easy to adapt your new code. I can also help you with that.

pminervini commented 6 years ago

Talking about NLP stuff.. I tried to go through/understanding https://github.com/uclmr/jack/blob/master/jack/util/map.py but failed miserably. Do we want to do something about it? Is this level of complexity even needed?

dirkweissenborn commented 6 years ago

haha, yeah that code is wild. It basically implements different kinds of nested map functions. It is rather a utility function than a core part of jack. it is rarely used in the current code base