search

openeventdata / Focus_Locality_Extraction

MIT License
2 stars 4 forks source link

Multiple docs/reloading MITIE #4

Open ahalterman opened 5 years ago

ahalterman commented 5 years ago

It seems from the test code that every time that Profile.main() gets called, once it gets to SentenceExtraction_test, it reloads the MITIE/Stanford model, which is obviously very, very slow. Am I reading that right? Is there either a way to pass a list of stories into Profile.main()? If not, it seems like it would make sense to load the NER once inside Profile.main() and pass it into SentenceExtraction_test with each sentence (same as the embedding model). Does that make sense? Am I missing an option?

Maryam-Imani commented 5 years ago

It seems from the test code that every time that Profile.main() gets called, once it gets to SentenceExtraction_test, it reloads the MITIE/Stanford model, which is obviously very, very slow. Am I reading that right? Is there either a way to pass a list of stories into Profile.main()? If not, it seems like it would make sense to load the NER once inside Profile.main() and pass it into SentenceExtraction_test with each sentence (same as the embedding model). Does that make sense? Am I missing an option?

You are right. Thanks for mention it here. I will change it.