search

bwarren2 / datadrivendota

Codebase for dota analytics
Other
0 stars 0 forks source link

Write JSON directly from java? #567

Closed bwarren2 closed 8 years ago

bwarren2 commented 8 years ago

Currently, the parserpipe is doing lots of work to accommodate my discomfort with java/comfort with python. In particular, right now:

  1. Python gets information about matches that need to come in, shallow imports it.
  2. Python hits node service for URL via request.
  3. Python passes a message to rabbitmq for the java engine to parse a file at that url
  4. Java parses one big json, puts it on s3, send that filename back to python
  5. Python takes the file and splits it into many pieces, making many requests to put stuff on s3.

It would be cleaner to just write to s3 directly from java.

bwarren2 commented 8 years ago

See also #568

wlonk commented 8 years ago

Owen suggests using Java's Jackson library for parsing/writing JSON. "It internalizes a lot of complexity."

bwarren2 commented 8 years ago

We are fast enough cutting various files as is.