search

openeventdata / UniversalPetrarch

Language-agnostic political event coding using universal dependencies
MIT License
18 stars 9 forks source link

Make UniveralPetrarch pipeline compatible #43

Open ahalterman opened 6 years ago

ahalterman commented 6 years ago

Now that UP is a little more stable, we need to start thinking about making it usable in production pipelines. In order for people (specifically the Spanish and Arabic teams) to be able to produce event data, UP needs to fit into our existing pipelines. This requires a few things:

  1. making UDPipe consume the JSON/Mongo format that the OEDA pipelines use, rather than XML
  2. writing custom code to fit UDPipe into, e.g. the stanford_pipeline. It should output OEDA-formatted JSON to store back in a Mongo.
  3. greatly simplifying the UDPipe installation process and ensuring that the correct versions are used.
  4. making sure UniveralPetrarch can take in JSON. I added code to do this (see test code here) but this should be tested with the actual output from 1 and 2.
PTB-OEDA commented 6 years ago

@JingL1014, didn't we already do already do this with Sayeed? Did that code get pushed back here yet?

ahalterman commented 5 years ago

Following up on this. Are any of these (1, 2, 3, 4) complete? This is necessary for us to produce Arabic event data.

Sayeedsalam commented 5 years ago

  1. We have updated UniversalPetrarch to consume JSON formatted data. The UD-Petrarch coder for English is running side-by-side with Petrarch2 event coder in the SPEC Pipeline.

  2. We are not using stanford-pipeline project to generate event coding from raw text data. Instead, we running our distributed framework. So we haven't tested adding UD-Petrarch to that pipeline.

  3. We currently use ufal-udpipe package for Python to do the parsing. https://pypi.org/project/ufal.udpipe. So use the parser, we need to install the package and download language-specific model files which can be automated. (i.e using requirements.txt and some programming)

  4. Yes, we are already using UD-Petrarch to code the English sentences and both input and output are in JSON format. Any incompatibility with OEDA format can be addressed. We are using MongoDB to store that data.