How to give test data as pdf instead of annotated json file

[chaitanya1019]

In this repo, u have given annotated test data for prediction, but what if the test data is not annotated before hand for prediction. I mean what modifications should be made to take input as a pdf not an annotated json file.

[chaitanya1019]

In this repo, u have given annotated test data for prediction, but what if the test data is not annotated before hand for prediction. I mean what modifications should be made to take input as a pdf not an annotated json file.

I've use pdfminier.six package to convert pdf to text and apply my custom ner model to make predictions. But the predictions are not coming out to be accurate. Out of the 9 entities(Companies Worked at Skills Graduation Year College Name Degree Designation Email Address Location Name) that the training data set has only 5-6 entities are recognized even after 100 iterations of 200 resumes. Please follow this https://github.com/explosion/spaCy/issues/3528#issuecomment-483947072

[srk86386]

I am getting this error when I try to run the script :

ValueError: [E024] Could not find an optimal move to supervise the parser. Usually, this means the GoldParse was not correct. For example, are all labels added to the model?

[thongtran957]

In this repo, u have given annotated test data for prediction, but what if the test data is not annotated before hand for prediction. I mean what modifications should be made to take input as a pdf not an annotated json file.

I've use pdfminier.six package to convert pdf to text and apply my custom ner model to make predictions. But the predictions are not coming out to be accurate. Out of the 9 entities(Companies Worked at Skills Graduation Year College Name Degree Designation Email Address Location Name) that the training data set has only 5-6 entities are recognized even after 100 iterations of 200 resumes. Please follow this explosion/spaCy#3528 (comment)

Hi @chaitanya1019 , you can test data with pdf file? if yes, can you share with me how to test.

[sayalraza]

Hi @chaitanya1019 Could you please share the code with which you are doing inference. How are you using pdf/doc/docx to test the model instead of already annotated testdata.json file?

[sayalraza]

@chaitanya1019 I was able to successfully give a pdf/txt file to model and get a decent output. I don't think your code is the problem. I had same issue of model being messed up after loading from disk. I could find out that it was an issue with older version of spacy. I am using spacy 2.2.3. I trained the model with this and was able to do inference without messing up the model. The only problem is the dataset. In the new version, they dont allow dataset entities to overlap and this will raise ValueError: [E103]. I had to mannually remove all conflicting entities from dataset as there is no particular pattern. This was a time consuming task. Anyways, I have the clean traindata.json and testdata.json with me. Not able to attach json here though. I can share it any other way, if you want.

[mhmadayad]

@sayalraza could you please send me the clean traindata.json and testdata.json to mohamad.ayad@tum.de. Thanks in advance.

[mhmadayad]

@sayalraza as I am working to generalize the model by adding thousands of labeled training data. so I have to take care about entity conflicts otherwise its gonna be really time consuming to clean a json file with 1000 entry

[Noorain99]

@sayalraza could you please send me the clean traindata.json and testdata.json to noorainzaidi99@gmail.com. Thanks in advance.

[hardikjamnal404]

Heyy! @sayalraza If you can please send the traindata.json and testdata.json to hardikjamnal@gmail.com Thank You :)

[ziodos]

@sayalraza hey can you please provide me the cleaned version of traindata.json and testdata.json. this is my email: zied.zanina14@gmail.com Thanks in advance

[puttapraneeth]

@sayalraza could you please provide the cleaned version of traindata.json and testdata.json. My email: puttapraneeth@gmail.com

Also what is causing conflict in the json provided. I am getting error as there is a conflict, unable to understand what is that. Thanks in advance

[JasonLing95]

@sayalraza please could you provide a cleaned version of train data.json for new version of SpaCy model. Email: jason_ling95@hotmail.com

Really really appreciated.

[swethasrinivasan16]

@sayalraza please could you send the cleaned version to mail: swethas162001@gmail.com

[KaranvirSIdana]

@sayalraza please could you send the cleaned version to mail: swethas162001@gmail.com

Have you got the cleaned version? If yes, could you please share it with me as well.TIA!