prashant334
commented
6 years ago @ines ANY UPDATE HOW TO RESOLVE THIS?
Closed prashant334 closed 6 years ago
prashant334
commented
6 years ago @ines ANY UPDATE HOW TO RESOLVE THIS?
SandeepNaidu
commented
6 years ago Prashant, train_data is empty? Did you not paste that code? Can you paste some sample?
prashant334
commented
6 years ago I intentionally keep it empty because they are too big. But here I am attaching train data sample.txt
SandeepNaidu
commented
6 years ago Try changing this line gold = GoldParse(doc, entities=entity_offsets) to gold = GoldParse(doc, entities=entity_offsets['entities'])
Later after fixing this, Also the label strings you are adding to the ner component are different from the sample text you have given. Make sure they match.
prashant334
commented
6 years ago @SandeepNaidu yes labels I have changed. Please review below code.
from __future__ import unicode_literals, print_function
import random
from pathlib import Path
import random
import openpyxl
import spacy
import spacy.pipeline
from spacy.gold import GoldParse, minibatch
from spacy.pipeline import EntityRecognizer
#from spacy.pipeline import TokenVectorEncoder
def get_gold_parses(tokenizer, train_data):
'''Shuffle and create GoldParse objects'''
random.shuffle(train_data)
for raw_text, entity_offsets in train_data:
doc = tokenizer(raw_text)
#gold = GoldParse(doc, entities=entity_offsets.get('entities'))
gold = GoldParse(doc, entities=entity_offsets.get('entities'))
yield doc, gold
def train_ner(nlp, train_data, output_dir):
random.seed(0)
optimizer = nlp.begin_training(lambda: [])
nlp.meta['name'] = 'CRIME_LOCATION'
for itn in range(50):
losses = {}
for batch in minibatch(get_gold_parses(nlp.make_doc, train_data), size=3):
docs, golds = zip(*batch)
nlp.update(docs, golds, losses=losses, sgd=optimizer,
drop=0.50)
print("under learning")
if not output_dir:
return
elif not output_dir.exists():
output_dir.mkdir()
nlp.to_disk(output_dir)
def main(model_name, output_directory=None):
print("Creating initial model", model_name)
nlp = spacy.blank(model_name)
#nlp.entity.add_label('CATCHPHRASE')
if output_directory is not None:
output_directory = Path(output_directory)
train_data = [
(
"Controversial self-styled Godman Sarathi Baba was arrested today by Odisha Police's Crime Branch for his alleged involvement in a sexual escapade as well as cases related to cheating and forgery. The arrest came after nearly 14 hours of intense grilling at his ashram at Barimula in Kendrapara district and later at Crime Branch headquarters at Cuttack. Crime Branch ADG BK Sharma confirmed his arrest on Saturday afternoon. He was produced in a court today, which has remanded him to a three day custody for further investigation. The self-styled Godman has been at the centre of a controversy after a TV channel showed pictures alleging that he had spent two days with a woman, who posed as his wife, at a Hyderabad hotel. Sarathi, however, denied the charge and threatened to take legal action against the TV channel. Sarathi has been booked under various sections of the Indian Penal Code including 420 (cheating), 120 (B) criminal conspiracy and 341 (wrongful restraint) as well under the Arms Act and SC/ST Act. A Crime Branch team, which reached Sarathi's ashram on Friday, questioned the Godman and conducted a thorough search of the sprawling premises, police said. Some documents, cash, silver, gold and metal coins with his photo engraved were recovered during search, the police added. The Crime Branch officers also came across some bank accounts, photographs and CCTV footage during the operation and these were being examined, the police said. Protests against Sarathi rocked Kendrapara town on Friday with people involved in pitched battles with the police. While prohibitory orders under Section 144 of CrCP were clamped in the area, SP Satish Gajbhiye was shifted from Kendrapara and Nitinjit Singh replaced him.",
{'entities': [(271, 302, 'CRIME LOCATION'), (673, 678, 'VICTIM')]}
),
(
"Aam Aadmi Party today targeted the Delhi Police saying it was wasting its energy by leveling false charges against party lawmakers and that the credibility of the force is under question owing to rising incidents of crime in the city. Addressing a press conference, AAP Delhi Convenor Dilip Pandey cited the arrest of a constable for allegedly gang-raping a woman in Ranhola area of West Delhi on Thursday night in countering the assault charges against party MLA Commando Surinder Singh. How would anyone feel secure in this city when men who are supposedly protectors end up committing such crimes? It's all because the police is busy cooking up false charges against our MLAs, Mr Pandey said. Mr Pandey blamed the police of conspiracy against the public and said that the trust deficit or the credibility crisis is purely because of the fact that police continue to work under the Narendra Modi government, adding that it should focus on women's security instead, otherwise it stands discredited.",
{'entities': [(369, 395, 'CRIME LOCATION'), (358, 363, 'VICTIM')]}
),
(
"A Sub-Inspector of police was today arrested here for allegedly misbehaving with a 32-year-old woman and her daughter, police said. M Satish Kumar, the officer, allegedly sought sexual favours from the woman who had come to the Medchal police station yesterday with some complaint. He misbehaved with her, and also misbehaved and abused her daughter later, Assistant Commissioner of Police S Ashok Kumar told PTI. A case under Section 354 (assault or criminal force to woman with intent to outrage her modesty) of IPC and under the Protection Of Children From Sexual Offences Act has been registered. The accused was arrested and produced before a court and sent in remand, the ACP added.",
{'entities': [(228, 235, 'CRIME LOCATION'), (83, 117, 'VICTIM')]}
),
(
"An Army jawan was beaten to death in Meerut after he tried to rescue a girl from harassment. Vedmitra Chaudhury, 35, a Lance Naik in 416 Engineering Brigade, had gone to get milk from a booth on Rohta Road in Hardevnagar on Thursday where he saw some men teasing the daughter of the shop owner. The jawan came to the girl's rescue and even beat up one of the accused named Akash, police said. Enraged, Akash reportedly called his friends, who arrived at the spot armed with sticks. They then allegedly assaulted the armyman, leaving him seriously injured. The jawan, who was staying at Gopal Vihar with his wife and children, was admitted to an Army Hospital in the area, where he succumbed to his injuries last night, police said. Senior Superintendent of Police Dinesh Chandra Dubey said, beside Akash, two others, Sanju and Ritesh, have been arrested and the search was on to arrest the other accused. Talking about the incident, Sub-Area Colonel Rajiv Kumar said that Mr Chaudhury fulfilled the duty of an Indian citizen and protected a girl's honour. He will be remembered as a martyr.",
{'entities': [(37, 43, 'CRIME LOCATION'), (3, 13, 'VICTIM')]}
),
(
"An elderly Indian-American couple has been shot dead in the US state of North Carolina and police have arrested a 20-year-old suspect in connection with the slayings. Kantibhai Patel, 72, and his wife Hansaben Patel, 67 worked and lived at Best Western Point South were shot and killed on Sunday morning. Hours later, the Jasper County Sheriff's Office arrested Joshua Lenard Poacher in connection with the slayings, Jasper County Sun Times reported. Poacher was charged with two counts of murder in the killings of the Patels, it said. Sheriff Greg Jenkins said Poacher was arrested without incident at a hotel in Ridgeland. The Patels lived at the Point South-based hotel and had worked there for 10 years, according to the sheriff's office. A beautiful couple, no doubt, Jenkins said. Lives were taken from them, senselessly. You could not give an account for such a tragedy; why an individual would do such a thing. Jenkins said they were able to track down Poacher after he tried to use credit cards taken from the Patels. Police also viewed surveillance footage and received tips via phone calls. It was not known what led to the homicides. Jenkins did not know if Poacher knew the couple. What led up to that, what was the relationship between the three individuals we have no idea Jenkins said. The Patels were found shot inside a room at the hotel. Jenkins said his office received tips that helped lead to Poacher's arrest, but there were no eyewitness accounts, no reports of an argument, shouting or a fight.",
{'entities': [(56, 86, 'CRIME LOCATION'), (3, 33, 'VICTIM')]}
),
(
"Refusing to serve a piece of lemon cost an eatery-owner his life in Bihar's Begusarai district on Monday. Four men, who had come to eat at 29-year-old Ratish Kumar's restaurant close to a prominent railway station late on Monday night, got into a tiff with the owner after they asked for lemon wedges along with their meal but were refused. The four youth then threatened the owner with consequences but left the eatery. However, they returned after an hour, close to midnight, with a pistol and fired at Mr Kumar from point blank range, killing him on the spot. The youth then fled on motorcycles before anyone could react. The restaurant is barely 100 metres away from a railway police station in the area. Following the incident, locals agitated at many places in Begusarai and blocked many roads for hours. The Bihar Police say they are taking steps to ensure arrests in the case soon. The four people, all of a young age, were frequent visitors to the eatery. We are trying to trace them and will make arrests soon, said Jitendra Mishra, a senior police official in Begusarai. Earlier this month, campaigning for upcoming elections in Bihar, Prime Minister Narendra Modi had said that the rising number of crimes in the state showed how Chief Minister Nitish Kumar's government was faring. He said there was a 47 per cent rise in murders in the first six months of the year alone.",
{'entities': [(68, 94, 'CRIME LOCATION'), (43, 55, 'VICTIM')]}
),
(
"A 53-year-old former teacher has been jailed for nearly four years for secretly filming over 100 teenage students taking shower at a private boarding school in the UK - totalling more than 2,500 hours of footage - filmed over a period of 16 years. Jonathan Thomson-Glover, pleaded guilty to 36 counts of making, taking and possessing indecent images of 120 children aged between 12 and 17 over a 16 year period while at Clifton College, Bristol. He was jailed for three years and nine months by Judge David Ticehurst at Taunton Crown Court yesterday. You are the author of your own misfortune and there can be little sympathy for you. It is impossible to calculate the harm and damage you may have caused to those who trusted you or were in your care, the judge said. Police found more than 300 video tapes from Jonathan's house that included more than 2,500 hours of footage, the Independent reported. The German teacher used holidays to fix cameras into the walls - linking them to video recorders back in his private quarters to record intimate moments of students in shower and in their bedrooms. The former teacher was arrested last year after UK's National Crime Agency found that his computer was being used to download inappropriate images of teenage children.",
{'entities': [(420, 444, 'CRIME LOCATION'), (2, 28, 'VICTIM')]}
),
(
"A 27-year-old Indian man has been sentenced to three months in prison for harassing a woman at a shopping mall in Dubai, a media report said today. The incident took place in Deira city on May 29 when the 35-year-old woman from the Philippines entered a lift and was harassed by the drunk labourer. The court also slapped a fine of $545 on the convict for consuming alcohol without a licence and has ordered his deportation. It happened near the loading area. He was standing by the lift, the woman was quoted as saying by the Khaleej Times. She claimed that the man groped her intentionally when she was about to enter the lift and was forced to walk out immediately after the incident while the man stayed in the elevator. The woman screamed for help and caught the attention of nearby security guards who arrived immediately to her rescue, the report said. The man tried to flee but was caught and handed over to police. The man admitted when questioned by the police that he had been in an inebriated state but claimed it was not an intentional attempt. The verdict can be appealed within 15 days.",
{'entities': [(95, 119, 'CRIME LOCATION'), (86, 91, 'VICTIM')]}
),
(
"The driver of a courier van was killed and a security guard injured by unidentified armed men who looted gold worth crores of rupees from the vehicle on the Lucknow-Kanpur National Highway here, police said today. The van of Sequel Logistic Private Limited, Ahmedabad was on its way to Lucknow when the men driving in an SUV intercepted it near Bajhera village, they said. They shot the driver Harichand Yadav, 30 and security guard, who offered resistance, and looted gold biscuits and jewellery weighing over seven kg before fleeing, they said. Another person, who was in the van, managed to save his life and rushed to inform the police, they said. The driver and the guard were rushed to the Nawabganj community health centre where the former was declared dead and the latter was referred to Lucknow Trauma centre in a serious condition, they said. A first information report has been filed on the complaint of the courier company for loot of over seven kg of gold in the form of biscuit and jewellery, one mobile phone and three ATM cards, they added.",
{'entities': [(153, 188, 'CRIME LOCATION'), (4, 59, 'VICTIM')]}
),
(
"With the arrest of two persons, including a woman and recovery of marijuana worth around Rs 1 crore, the Crime Branch of Delhi Police today claimed to have busted an inter-state drug module. The accused, Rahul (22) and the 29-year-old woman, whose name was not disclosed, were arrested from near Anand Vihar ISBT, where they had come to meet an associate, said police. The duo had arrived there in a car and a marijuana consignment weighing around 1 Kg was concealed inside. The consignment was procured from Odisha, said Joint Commissioner of Police (Crime) Ravindra Yadav. They turned out to be delivery persons working for the module and earned around Rs 10,000 for each delivery, said Mr Yadav. During interrogation, they told police that they were working for an alleged drug trafficker, Manjeet, who is presently lodged in Tihar jail. Manjeet's associate at Odisha, identified as Mama, is presently heading the module, procuring marijuana from Lamtaput and other naxalite areas in Odisha, said a police officer. In a separate incident, a Police Control Room team patrolling at outer Delhi's Mongolpuri area recovered around 6.20 kg marijuana from the possession of a youth, police said today. The accused, identified as Santosh (20), tried to flee on seeing the PCR van. He was arrested after a brief chase, said Additional Commissioner of Police (Operations).",
{'entities': [(296, 312, 'CRIME LOCATION'), (0, 0, 'VICTIM')]}
),
(
"An eld{'entities': [(131, 138, 'CRIME LOCATION'), (3, 28, 'VICTIM')]}erly Sikh-American man was brutally injured and allegedly called a terrorist and Bin Laden in an apparent hate crime case in Chicago, just days before the US commemorates the anniversary of the September 11 attacks. Inderjit Singh Mukker was assaulted on Tuesday when the accused allegedly pulled up to his car yelling racial slurs, including, Terrorist, go back to your country, Bin Laden! Mr Mukker, a US citizen and father of two, was on his way to a grocery store and was repeatedly cut off by a driver. He pulled over to the side of the road to let him pass but the driver instead pulled in front of his car and aggressively approached Mr Mukker's vehicle, according to information by the Sikh Coalition, a community-based organisation said. The accused then allegedly reached into the car and repeatedly punched Mr Mukker in the face, causing him to lose consciousness, bleed profusely and suffer a fractured cheekbone and a laceration to his cheek. Mr Mukker was rushed to a nearby hospital where he received six stitches, treatment for lacerations, bruising and swelling. The suspect is in custody. No American should be afraid to practise their faith in our country, Mr Mukker said. I am thankful for the authorities swift response to apprehend the individual but without this being fully investigated as a hate crime, we risk ignoring the horrific pattern of intolerance, abuse and violence that Sikhs and other minority communities in this country continue to face. The Sikh Coalition's Legal Director Harsimran Kaur said the group believes that Mr Mukker was targeted and assaulted because of his Sikh religious appearance, race or national origin. We request an immediate investigation and call on local and federal agencies to investigate this attack as a hate crime, Ms Kaur said. Sikh Coalition said the attack is just the latest in a line of similar violent assaults on Sikhs in America. Last August, Sandeep Singh, a Sikh father in New York City, was run over and dragged 30 feet after being called a terrorist. In 2012, a gunman walked into a Sikh house of worship and shot and killed six innocent Sikh victims in Oak Creek, Wisconsin.",
{'entities': [(131, 138, 'CRIME LOCATION'), (3, 28, 'VICTIM')]}
),
(
"An American man has been sentenced to 13 years in prison on charges of hate crime for brutally hitting an 82-year-old Sikh with a steel rod outside a gurdwara in the US state of California two years ago. Piara Singh had just finished preparing meals for the lungar (community kitchen) when Gilbert Garcia spotted him outside a Southwest Fresno gurdwara. Mr Garcia screamed something about the Taliban and Muslims as he attacked Mr Singh. Local prosecutors treated the incident on May 5, 2013 as a hate crime. Mr Singh was hospitalised for more than a week with head wounds, broken ribs and lung lacerations. It stands to reason from common sense that anyone who would engage in this type of horrific behaviour would be someone who is classified as not necessarily being normal, said Fresno County Superior Court Judge Alvin Harrell III. The bottom line is justice was meted out today and in our community, if you are going to commit a hate crime, you are going to go to prison, said prosecutor Tim Donovan. Justice has been served. Not only does the victim and his family have closure, but so does the entire Sikh community because there was an acknowledgement that a hate crime occurred, Ike Grewal a local Sikh community activist told local media outlets. According to Mr Grewal, Gilbert Garcia made racial slurs about Muslims and told Mr Singh: I am going to destroy your mosque. Sikhs he said, continue to be mistaken as terrorists because the media tends to show instances of terrorist attacks and images of the terrorists wearing turbans. ",
{'entities': [(148, 188, 'CRIME LOCATION'), (106, 122, 'VICTIM')]}
),
(
"Two persons have been arrested by the city Crime Branch for allegedly attacking a man who was a follower of controversial self-styled godman Asaram, but later turned against him, police said today. We have arrested Basav Raj and Sheetal Prajapati in the case of firing on Asaram's former follower Raju Chandak in 2009 in Chandkheda area of the city, a Crime Branch official said. Mr Chandak, after the mysterious death of two cousins Dipesh and Abhisek Vaghela of Asaram boarding school, had distanced himself from Asaram Bapu. He had later deposed before Justice DK Trivedi Commission which was appointed to probe the mysterious death of the two boys. Police is also investigating the role of these two arrested persons in connection with the attacks on witnesses of rape cases filed against Asaram Bapu and his son Narayan Sai, the official said.",
{'entities': [(321, 348, 'CRIME LOCATION'), (82,85,'VICTIM')]}
),
(
"Two days after a 16-year-old girl was found dumped in an empty well after allegedly being raped and shot at in Greater Noida, three men were detained on Monday for their suspected role in the crime. While one of the three accused is a resident of Ghaziabad, the other two live in Greater Noida. They are aged between 18-20 years, said a police official. The minor girl was allegedly abducted from west Delhi's Mohan Garden area, raped for around 13 days and shot at by the accused when she tried to escape from a room in Greater Noida where she was held captive, he said. One of the suspects was known to the girl and he is believed to have abducted her on November 22 when she went out to purchase milk. Preliminary investigation also revealed that the girl talked to the suspect over phone the previous night, police said. A case of abduction was reported in Uttam Nagar police station after the girl went missing by her family. Giving more details of the crime, the police said that the three accused took the girl from one location to another in Greater Noida, where they allegedly raped her for around 13 days. When she tried to escape on Saturday, the three accused caught her and shot her. The bullet hit her chest and the girl was dumped in an empty well in the village, police said. She was later rescued by residents of Salempur Gurjar village in Greater Noida, who also informed the police about the incident. The police stated that the girl was recuperating and her statement has been recorded. We are questioning the suspects and the entire case is likely to be transferred to Delhi Police, said DCP (West) Pushpendra Kumar. Police noted that there were a lot of discrepancies in the versions of the victim, her family and the suspects in the case. All have to be cross-examined before they arrive at a conclusion, they said. We are yet to find the weapon used in firing at the victim, they added. A case has also been registered at Greater Noida police station under relevant sections of the law in the incident, police said.",
{'entities': [(111, 124, 'CRIME LOCATION'), (15, 33, 'VICTIM')]}
),
(
"A Gurdwara has been vandalized in Los Angeles suburbs with hateful graffiti addressed towards ISIS, Sikh community leaders have said, fearing this hate crime is a repercussion of the California shooting. We are concerned about the safety and security of our community members. We are of the opinion that this is a hate crime and that this is a direct result of a possible backlash from the San Bernardino killings, said Inderjot Singh, president of Sikh Gurdwara in Buena Park, a suburb of Los Angeles. The incident, which occurred early on Sunday morning is being investigated by local law enforcement agencies. The Gurdwara was vandalized during early hours of Sunday and a hateful graffiti was seen on the walls of the Gurdwara and also a truck parked in the parking lot. The graffiti included the phrase, F@#k ISIS, and the words Islam and other reference of gangs, Washington-based Sikh Council on Religion and Education said in a statement. A hateful graffiti was seen on the walls of the Gurdwara and also a truck parked in the parking lot. The Gurdwara is attended by over 800 community members on a weekly basis. The White House, which was informed about the incident, has forwarded the incident to the Department of Homeland Security to investigate. The Sikh community across the nation is in a heightened state of alert and is deeply troubled by this latest incident. We are appealing to all Sikh place of congregation to be in touch with the local law enforcement agencies as well with the elected officials, Dr Rajwant Singh of the Sikh Council on Religion and Education, said. Dr Singh also expressed concern over the recent anti-Muslim rhetoric made by some of the contenders of the American presidency. This will directly result in elevated level of violence against minority religions in America and particularly the Sikhs. We are fearful that this kind of hate speech against Muslims with engulf Sikhs and members of the Muslim community across the nation, he said. ",
{'entities': [(34, 45, 'CRIME LOCATION'), (2, 10, 'VICTIM')]}
),
(
"The Delhi Police of crime branch with the help of local police has busted a gang which was allegedly involved in the trafficking of girls, and arrested three persons in this connection at Goela village under Shahpur police station in the district. According to police, a team of Delhi Police led by Deputy Commissioner Rajiv Sharma raided a house yesterday and recovered a girl belonging to Jharkhand, who went missing from Delhi on November 21. The girl was allegedly abducted on the promise of providing her a good job, but was allegedly sold to a villager Sandeep, who kept the girl in her house. Police also recovered another 19-year-old girl who was allegedly sold for Rs 40,000 to a villager Narenderby one Arvind. The three accused persons, Sandeep, Arvind and Narender have been arrested, police said. During interrogation, it came to the fore that Arvind and his wife Anjali were 'active' in supplying missing girls on the pretext of providing them good jobs. Anjali is absconding. Seven missing girls have been identified who were allegedly sold to different people, they added.",
{'entities': [(188, 201, 'CRIME LOCATION'), (132, 137, 'VICTIM')]}
),
(
"Two persons, including a senior citizen, were allegedly shot at before being robbed of cash in two separate incidents which took place within two hours in northeast Delhi on Monday. In the first incident, which took place around 9 am, a 70-year-old man was allegedly shot at by robbers who made away with Rs 3 lakh cash he was carrying, said a police official. The victim, Jagbir Singh, who works for a cooking gas supplier, was heading towards office in Gokulpuri with an associate when the robbers waylaid them. When the man tried to resist, they allegedly fired two rounds, of which one bullet hit the elderly man's leg, and fled with the bag containing the cash, said the official. In the second incident, which took place around 11 AM, a man identified as Rajbir Singh was heading towards Gandhi Nagar from his residence in Maujpur, when two bike-borne persons waylaid him midway and tried to snatch his bag containing around Rs 10 lakh cash, which Singh was going to deposit in a bank, said a police official. When he tried to resist, the robbers allegedly fired and the bullet hit his arm, following which Mr Singh collapsed on the ground and the robbers fled with the case, the official said. The victims in both the cases are admitted in a hospital and their conditions are reported to be stable. Cases have been registered and separate teams have been constituted to investigate both the cases, a senior official said.",
{'entities': [(155, 170, 'CRIME LOCATION'), (237, 252, 'VICTIM')]}
),
(
"The police today arrested four people in connection with the murders of two engineers of a private road building company who were shot dead in Bihar last week. Another engineer was found murdered in the state on Monday. Ankit Jha was working as a quality engineer with Reliance Telecom. His body was found on Monday morning with his throat slit in Vaishali district, 60 km from capital Patna. The killings are fast becoming a political issue with the opposition calling it the return of the law of the jungle in Bihar - a phrase used to describe incidents of lawlessness during the tenure of Lalu Prasad Yadav's RJD from 1990 to 2005. The party is now part of the ruling coalition led by JDU's Nitish Kumar. Jungle Raj (lawlessness) is back in Bihar. In the run-up to Assembly polls, we were saying that there will be Jungle Raj-2 in Bihar if Nitish-Lalu combine wins the elections, Union Minister and LJP chief Ram Vilas Paswan of the BJP-led alliance that lost the assembly elections in October and November said. An engineer's body was found in Vaishali yesterday, two engineers were butchered in Darbhanga, prior to that a business man was shot in Muzzfarnagar, a Sarpanch was killed in my parliamentary constituency in broad day light, Union minister and BJP leader Rajiv Pratap Rudy said. Rajiv Jha, brother of the Reliance engineer, said Mr Jha had left for work on Sunday night. He left at night saying his company's car was waiting outside. I thought it must be office work. When I tried to call him in the morning, both his phones were off. We then started looking for him but couldn't find him. Then we got a call from the police, he said. In the Darbhanga murders, apart from todays arrests, a police official was suspended and the investigations handed over to a Special Task Force. The police suspects the two engineers who were killed on Saturday were targeted after the company they worked for failed to meet a ransom demand. Sources in the police have however ruled out extortion as the reason behind Mr Jhas murder. Project engineers Brajesh Kumar and Mukesh Kumar were killed while they were supervising construction of a road at the Darbhanga-Kusheshwarsthan highway. The police had deployed personnel at the construction site after their company had received a ransom call. But the policemen were removed from the spot on Friday for local elections and were not present when the attack happened.",
{'entities': [(348, 365, 'CRIME LOCATION'), (72, 85, 'VICTIM')]}
),
(
"A woman was allegedly raped, filmed and threatened at the Infosys campus in Pune. Two men have been arrested for the crime which took place on Sunday, police said. The 25-year-old woman and the suspects worked at the canteen of the outsourcing major, said the police.The woman had gone to washroom to freshen up. She was allegedly raped there. Yesterday she came to the police station to file her complaint, said Inspector Vishwajeet Khule. He added that in her complaint, the woman said she had been filmed during the crime and blackmailed by the suspects. The two accused have been sent to police custody till the January 1 and the police are waiting for the medical reports of the woman. In a statement, Infosys said, We are investigating the matter and are working closely with the police to aid their investigation. We have stringent security measures in place on all our campuses and have a zero tolerance policy against sexual harassment which extends to our contractual staff as well.",
{'entities': [(76, 80, 'CRIME LOCATION'), (2, 7, 'VICTIM')]}
),
(
"A 68-year-old Sikh man was stabbed to death in California's Fresno city on January 1, prompting police to launch a hate crime probe into the city's first homicide. Gurcharan Singh Gill, an employee at local liquor store at Shields Express in West Shields Avenue was at work when he was stabbed to death during broad day light, Lieutenant Mindy Casto said. While investigations are still going on, local community members feel Mr Gill was attacked due to his identity and skin colour, officer Casto said. When officers of Fresno Police Department arrived at the scene they found an elderly man lying in the shop. Emergency medical team tried to provide first-aid but it was determined that Gill had already died, Fresno Bee reported. The motive and cause of death are not known yet. His death marked the city's first homicide of 2016, police said. On December 28, another elderly Sikh man was brutally assaulted by two persons in Fresno. No arrests have been made in either cases. The police have appealed to the local people to report if they find any leads to the incident.",
{'entities': [(47, 71, 'CRIME LOCATION'), (2, 22, 'VICTIM')]}
),
(
"A minor girl was allegedly raped by a youth in the city and the crime came to light when she was involved in a road accident and found to be pregnant. The pregnancy of the 14-year-old girl was revealed when she met with a road accident and had to be admitted to hospital, police said. The girl, her parents and younger sister lived on a footpath in the Wagle Estate locality of the city and collected and sold scrap material to earn a livelihood. On January 6, the girl was knocked down by a vehicle on the road. She suffered injuries in the mishap and was rushed to a hospital in Mulund. A case was then registered with the Shrenagar police station, Thane police Public Relations Officer (PRO) Gajanan Kabdule said. Later, as the girl complained of severe stomachache, she was shifted to Sion Hospital where on Monday the doctors conducted sonography which revealed that she was one-month pregnant, he said. The doctors took the girl into confidence and probed further when it was revealed that she was allegedly raped by a 23-year-old Mehboob Sayeed, who used to target her at the Indira Nagar footpath, police said. Subsequently, the youth, who did petty jobs like electrical wiring, was arrested yesterday and booked under section 376 (rape) of the Indian Penal Code and sections 4, 8 and 12 of the Protection of Children from Sexual Offences (POCSO) Act 2012, police added.",
{'entities': [(2, 22, 'CRIME LOCATION'), (2, 7, 'VICTIM')]}
),
(
"Two days after a 16-year-old girl was found dumped in an empty well after allegedly being raped and shot at in Greater Noida, three men were detained on Monday for their suspected role in the crime. While one of the three accused is a resident of Ghaziabad, the other two live in Greater Noida. They are aged between 18-20 years, said a police official. The minor girl was allegedly abducted from west Delhi's Mohan Garden area, raped for around 13 days and shot at by the accused when she tried to escape from a room in Greater Noida where she was held captive, he said. One of the suspects was known to the girl and he is believed to have abducted her on November 22 when she went out to purchase milk. Preliminary investigation also revealed that the girl talked to the suspect over phone the previous night, police said. A case of abduction was reported in Uttam Nagar police station after the girl went missing by her family. Giving more details of the crime, the police said that the three accused took the girl from one location to another in Greater Noida, where they allegedly raped her for around 13 days. When she tried to escape on Saturday, the three accused caught her and shot her. The bullet hit her chest and the girl was dumped in an empty well in the village, police said. She was later rescued by residents of Salempur Gurjar village in Greater Noida, who also informed the police about the incident. The police stated that the girl was recuperating and her statement has been recorded. We are questioning the suspects and the entire case is likely to be transferred to Delhi Police, said DCP (West) Pushpendra Kumar. Police noted that there were a lot of discrepancies in the versions of the victim, her family and the suspects in the case. All have to be cross-examined before they arrive at a conclusion, they said. We are yet to find the weapon used in firing at the victim, they added. A case has also been registered at Greater Noida police station under relevant sections of the law in the incident, police said. ",
{'entities': [(111, 124, 'CRIME LOCATION'), (17, 33, 'VICTIM')]}
),
(
"A 28-year-old Sanskrit lecturer was brutally killed, allegedly by his younger brother in north Delhi's Burari area after a heated argument, police said today.",
{'entities': [(89, 31, 'CRIME LOCATION'), (17, 33, 'VICTIM')]}
),
(
"Three days after a teenager was killed by a speeding Mercedes near his home in Delhi's Paschim Vihar, Savneet Singh, a businessman from the same area has been arrested by the Crime Branch. 17-year-old Atul Arora was killed on Sunday, a day before he was to begin Class 12. He was driving a scooter, which was dragged 100 metres by the Mercedes before the driver sped away. The car has been seized, a senior police officer informed. The police suspect that the 27-year-old businessman was drunk when he rammed the boy's scooter head-on. He was allegedly with a friend and they were heading to an eatery when they hit Atul's scooter. The teen died instantly. He was not wearing a helmet and didn't have a driving licence, the police said. The impact of the crash was such that the boy was flung in the air and suffered severe head injuries. The car dragged Atul's scooty for at least 100 metres. The accused also hit a truck after killing Atul, said Suresh Arora, Atul's uncle. The police recovered the logo of the car, grill among other accessories from the accident spot, which suggested that the boy's scooter was hit by a luxury vehicle. They also identified the make of the car from the CCTV footage gathered from the spot, but couldn't see the number or the driver. Atul had wanted to travel the world and was extremely religious. He was returning from a satsang that night and had gone to drop a friend home when the accident happened, said Ruchi Arora, Atul's sister.",
{'entities': [(79, 100, 'CRIME LOCATION'), (19, 27, 'VICTIM')]}
),
(
"A 75-year-old man and his son have been arrested for allegedly running a betting racket in the garb of astrology business in southeast Delhi's Sangam Vihar area. The accused, Ram Snehi Lal, is a man with disability and had included his son (35) in the racket, allegedly being run by him, police said. Lal pretended to be an astrologer and was running the racket from his home. During previous raids, he took advantage of his disability and managed to claim innocence, police said. His son, Chotte, has also been arrested along with three alleged gamblers Ram Singh, Ravinder and Shyam Lal Verma, they said, adding that over Rs 15,000 were recovered from the three alleged gamblers. Police have also seized a register in which the father son duo used to keep a record of their clients and their betting amount. The racket was busted after an input was received by police and a raid was conducted yesterday around 9:30 pm. Initially he tried to mislead police by claiming that people were visiting him to consult him for their problems.",
{'entities': [(125, 160, 'CRIME LOCATION'), (2, 17, 'VICTIM')]}
),
(
"A 15-year-old boy was injured with glass shards by a group of young men in East Delhi's Farsh Bazar area for asking them not to throw water balloons at him, police said. The teenager was returning from his tuition classes when the a group of juvenile boys threw the balloons at him. When he protested and hurled abuses at them, they attacked him with broken bottles and glasses lying nearby, police said. A case of attempted murder has been registered against the accused and they have been apprehended, police said. The victim has told the cops that the incident occurred around 6:15 pm on Friday while he was returning from tuition classes with one of his friends on a scooter when they were waylaid by a group of boys near the Bihari colony in Farsh Bazar. When his friend protested, one of the accused walked up to them and tried to push him off the scooter. The victim, sitting pillion, got off the scooter and tried to intervene when the youth started thrashing him as well. My friend tried to intervene but they (the accused) threatened him with dire consequences, he told police. Family members of the victim said he has been preparing for his board exams, which are round the corner, and had planned not to participate in the festivals this year. The victim had lost his father recently and stays with his uncles and other family members at a colony near Farsh Bazar.",
{'entities': [(75, 104, 'CRIME LOCATION'), (2, 17, 'VICTIM')]}
),
(
"A drunk man allegedly thrashed his 75-year-old father to death over the father not storing water in southwest Delhi's Bindapur, police said. The deceased Ramkumar had failed to fill the water tank on Friday evening. When he told 40-year-old Chetan since he was not feeling well he could not do so, his son got angry and allegedly rained blows on him, police said. Many of the neighbours who heard the old man's cries did not come to his rescue and those who intervened were allegedly warned against it by the inebriated son, police added. Ramkumar, a line-man from MTNL, owned two houses in Khushi Ram Park. One of the houses was occupied by Chetan. After Ramkumar's wife died and his son's wife left him, he was made to do all the household chores and was thrashed if he did not agree. Chetan, an e-rickshaw driver, returned home around 10 pm on Friday and was watching television. Chetan's 14-year-old daughter, Tanya, who is an eyewitness, told police that he sat inside his room and poured liquor in his glass. He asked her to get him a glass of water, but she told him that there was no water at home. Chetan went to his father's room and started shouting. When Chetan's father opened the door, he asked him for water so he could add it to his liquor. The old man said that he could not fill the tank since he was unwell. He also said the water that was stored was for household usage and not for adding to liquor, police said. Chetan allegedly picked up a stick and started beating Ramkumar. The old man collapsed and died on the spot. Residents of the area told police that there had been no water supply in the area for the last couple of days. We registered a case under sections of murder soon after we received a complaint and the man was arrested from his house,said Surender Kumar, DCP (southwest).",
{'entities': [(100, 126, 'CRIME LOCATION'), (35, 53, 'VICTIM')]}
),
(
"A 30-year-old woman died after she was set on fire by her husband's relative for resisting rape in Korba district, police said today. The incident occurred last morning in Urga police station area, around 15 kms away from Korba, when Ramesh Patel (25), a cousin of the victim's husband, tried to force himself upon her when nobody was present at home, said the Urga police station SHO MM Minj. However, as the woman fought back, Ramesh thrashed her before dousing her in kerosene and setting her ablaze. On hearing her screams, the neighbourers rushed to her house and admitted her in a local hospital with around 70% burns. We managed to record her statement before she succumbed last evening, the SHO said. The officer said Ramesh had tried to rape the woman on February 22 as well but beat a retreat after she ward him off. He is a native of Madwarani village. A case has been registered in this connection and efforts are on to arrest the absconding accused.",
{'entities': [(99, 113, 'CRIME LOCATION'), (2, 19, 'VICTIM')]}
),
(
"HighlightsAccused lost his temper, brutally beat his first wife to death He then beheaded her with a saw and kept her head in a bag under the bed He had intended on getting rid of the body but was arrested The 40-year-old plumber, who has been arrested for brutally killing his wife in East Delhi's Madhu Vihar area, had lost temper due to extra spicy meal served to him after which he killed her, police said on Wednesday. It has also been found that the accused, Subodh, who had hacked into pieces the body of his first wife -- Manisha -- on the intervening night of February 11 and 12 used to have frequent quarrels with her over his second marriage. Police said they have found that Manisha had sustained 22 wounds as a result of being beaten brutally by Subodh on that night. A few months ago, Manisha had discovered that Subodh had married Muniya and thereafter they had frequent quarrels. She had left him to go to her relative's place in Rohini but on February 10 Subodh asked her to come to Madhu Vihar since he had planned to kill her, police added. Manisha got angry on seeing Muniya with Subodh but he told her that Muniya is staying in a room nearby, police said. That day when Subodh had got Manisha to his Madhu Vihar home, she cooked food for him and he found it too spicy, following which he beheaded her with a saw and kept the head in a bag in the bed-box. Police is speaking to family members to piece the sequence of events. ",
{'entities': [(286, 315, 'CRIME LOCATION'), (53, 63, 'VICTIM')]}
),
(
"A semi-decomposed body of a small-time local actress of the Bengali entertainment industry was today recovered from her flat in the southern part of Kolkata's Kasba area, police said. Bitasta Saha's semi-decomposed body was found hanging from the ceiling of her flat in Kasba area, they said. After repeated phone calls went unanswered for almost two days, her mother visited the flat today where she used to stay alone. When Ms Bitasta did not open the door after several knocks, her mother and neighbours informed police. Police broke the door and found the actress's body hanging from the ceiling, a police officer said. It was learnt that one of Ms Bitasta's wrist was slit and there were several injury marks on her body too. Going by the preliminary signs it seems that she committed suicide. However, we are waiting for the postmortem report. The body appeared to be two-day-old and there was a bit of stench in the room, he said. Police claimed the actress was under severe stress for quite sometime which could be a reason behind the incident. We have gone through her Facebook profile and posts, and found quite depressing posts on her wall... In fact at one point she had also talked about putting an end to her life... We are checking everything including her call details, police said.",
{'entities': [(128, 169, 'CRIME LOCATION'), (184, 196, 'VICTIM')]}
),
(
"man was booked for attempted murder after she was accused by her husband of throwing their two-and-half-year-old child from the staircase of their house in southeast Delhi's Pul Prahladpur area, police said on Thursday. On January 24, a PCR call was received around 6 pm wherein a man alleged that a child had been thrown from the staircase of house from first floor, a police official said. Police reached the spot and took the complaint from the man, who also told them that he has a video recording of the said incident, he said. The child was medically examined at AIIMS trauma centre on the same day and some some blunt injuries were found on the face, the official said, adding the victim's father owns a cosmetic shop in Sharma Market of Pul Prahladpur area. Police is investigating whether the couple were having marital issues over the child, he said. The officer also said that it is not known yet what was the provocation or motive of the mother to behave like this with her child. Police is also verifying the authenticity of the video recording and investigating whether these were regular episodes or a one-off incident and family members are being interrogated.",
{'entities': [(156, 193, 'CRIME LOCATION'), (91, 118, 'VICTIM')]}
),
(
"A man was allegedly robbed by three persons in Shahdara after they offered him a ride in their car, police said on Sunday. On Friday, Kapil Chowdhury (22) was waiting at Kashmere Gate for a bus to go to Ghaziabad when three men asked him for the direction to reach there and offered him a ride, a police official said. When the vehicle reached a secluded spot in Shahdara, they allegedly threw Mr Chowdhury out of the car after thrashing him and robbing him of his belongings, police said. The victim made a PCR call after borrowing a phone from a passerby, they said, adding the three accused were detained yesterday. In a similar incident, a cab driver was robbed of his belongings, thrashed and thrown out of his car yesterday after three persons boarded it on the pretext of going to Gurgaon. After boarding the cab, one of the accused pointed a pistol at him and asked him to drive on. On reaching Pul Prahladpur, took away Satyendra Kumar's belongings and threw him out of the car, police said, adding a hunt is on the nab the accused.",
{'entities': [(203, 212, 'CRIME LOCATION'), (134, 149, 'VICTIM')]}
),
(
"In a horrific incident, a 15-year-old boy allegedly killed schoolmate and chopped off his body into pieces in Ludhiana, police said today. The eight-year-old victim, Deepu, and the accused - were studying in the same school and belonged to adjoining villages in Uttar Pradesh. During the preliminary investigations, the juvenile revealed that he took Deepu to his migrant house in Ludhiana on the pretext of flying a kite last week and strangled him, Deputy Commissioner of Police Bhupinder Singh said. The juvenile then chopped off his body into six pieces and packed his body in two gunny bags and threw in a vacant plot at Dugri road. He removed the heart from the victim's body and threw the organ inside the premises of his school as he wanted to defame the institution, the officer said. The accused told the police that he had a grudge against the school. The family of the accused told the police that he was fond of eating fresh flesh of animals and on several occasions, chew his own skin, the officer said. He said that investigations of this sensitive case were under progress. The weapon used in the crime was also recovered.",
{'entities': [(110, 118, 'CRIME LOCATION'), (26, 41, 'VICTIM')]}
),
(
"A 22-year-old teacher was allegedly raped by the headmaster of a government-run school at Satwas town in Dewas district of Madhya Pradesh, police said today. The headmaster, Babu Khan Banarasi, has been arrested for the alleged crime that took place on the school premises on Wednesday evening, police said. The victim, a guest faculty teacher in the school, lodged a police complaint on Thursday and the accused was arrested today, police said adding, he was produced in the court, which sent him to judicial custody. Banarasi, the headmaster of a government middle school in Satwas town, was booked and arrested under the rape charge following a complaint made by a guest faculty teacher, Superintendent of Police (SP) Dewas district, Anshuman Singh told over phone. In her complaint, the victim said that Banarasi had asked her to stay back after the school hours saying he wanted to assign some official work to her. However, he sexually assaulted her in his office, the complainant said. Based on her complaint, the headmaster was arrested, Mr Singh said. The court has sent the accused to judicial custody. We are also sending the information about the headmaster's arrest to the state school education department, he added. There were media reports that the case was registered after a purported video of the incident went viral on social media platforms. The media reports claimed that a student playing on the school premises had recorded the crime on his mobile camera. When asked, the police, however, said they did not receive any such video. We have got this information from media reports. But the complainant did not mention about it, the SP added. Further investigation into the case is on.",
{'entities': [(90, 137, 'CRIME LOCATION'), (2, 21, 'VICTIM')]}
)
]
ner = EntityRecognizer(nlp.vocab)
labels = ['CRIME LOCATION', 'VICTIM']
for label in labels:
ner.add_label(label)
#nlp.add_pipe(EntityRecognizer(nlp.vocab))
#nlp.pipeline[nlp.pipe_names.index('ner')][1].add_label('CRIME LOCATION')
train_ner(nlp, train_data, output_directory)
# Test that the entity is recognized
wb = openpyxl.load_workbook('/home/prashant/Downloads/clsr500.xlsx')
sheet = wb.get_sheet_by_name('Sheet1')
for rowOfCellObjects in sheet['A2':'A30']:
for cellObj in rowOfCellObjects:
print("------------>>")
print(cellObj.value+":")
doc = nlp(cellObj.value)
for ent in doc.ents:
print(ent.label_, ent.text)
#words = []
#words.append("Yogaras Poongavanam was also sentenced to 15 strokes of the canes for trying to smuggle diamorphine into Singapore three years ago.")
text = "Man Arrested In Connection With Killing Of Senior Citizen In Delhi's Lajpath Nagar."
print("Man Arrested In Connection With Killing Of Senior Citizen In Delhi's Lajpath Nagar:")
doc = nlp(text)
for ent in doc.ents:
print(ent.label_, ent.text)
if output_directory:
print("Loading from", output_directory)
nlp2 = spacy.load(output_directory)
doc2 = nlp2('Do you like horses?')
for ent in doc2.ents:
print(ent.label_, ent.text)
main('en')@SandeepNaidu gold.pyx key error is not now showing. But ner results are zero. Here I am attaching my test data.
prashant334
commented
6 years ago @ines @honnibal What could be the reason for gold.pyx error?
ines
commented
6 years ago As @SandeepNaidu mentioned above, the labels in your code are not consistent. In some places, you're using CRIME LOCATION, and in others, it says CRIME_LOCATION. If your results are bad, this can also have other reasons: for example, the task might just be difficult to learn, especially with just a handful of examples.
Crime locations and victims aren't very well-defined, independent categories. They're mostly defined by the relationships and dependencies within the text. For example, if "California" refers to the US state, it's always a GPE entity. But is it a crime location? That really depends. Sometimes you can find that information in the same sentence. Sometimes not. That makes those labels very difficult to predict from machine learning point of view. A better strategy for training a model could be to focus on improving the predictions of PERSON, GPE, LOC etc. and then use the dependency parse (or a custom parser) to extract whether they refer to a crime location or a victim.
By the way, for general usage questions like this that are specific to your code, you might want to create a post on StackOverflow instead. This will reach more people and you'll often get quicker replies. On the issue tracker, we mostly try to focus on bug reports, performance issues and feature requests. It's also always helpful if you can provide a small code snippet that illustrates the problem, instead of dumping an entire script. This makes it easier for us and others to reproduce the problem and help.
prashant334
commented
6 years ago @ines I have used CRIME_LOCATION only in nlp.meta['name']. Another question is how to assign 2 entities for meta.
lock[bot]
commented
6 years ago This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
@ines @honnibal below is a code that I have used as u mentioned in my previous question. https://github.com/explosion/spaCy/issues/2440
after execution getting error like
Traceback (most recent call last): File "/home/prashant/PycharmProjects/sales_ag/AG_NER.py", line 118, in
main('en')
File "/home/prashant/PycharmProjects/sales_ag/AG_NER.py", line 86, in main
train_ner(nlp, train_data, output_directory)
File "/home/prashant/PycharmProjects/sales_ag/AG_NER.py", line 32, in train_ner
for batch in minibatch(get_gold_parses(nlp.make_doc, train_data), size=3):
File "/home/prashant/.local/lib/python3.6/site-packages/spacy/util.py", line 393, in minibatch
batch = list(cytoolz.take(int(batch_size), items))
File "/home/prashant/PycharmProjects/sales_ag/AG_NER.py", line 19, in get_gold_parses
gold = GoldParse(doc, entities=entity_offsets)
File "gold.pyx", line 418, in spacy.gold.GoldParse.init
KeyError: 0