Closed dolsysmith closed 3 years ago
Updated spark_utils.py so that the original JSON (unparsed) is stored in the tweet
field. This method avoids discrepancies arising from the difference between Python's and Spark's handling of null fields in the original tweet JSON. The short version is that some fields in Twitter's JSON schema are present with null
values, and others can be absent entirely. Spark, being schema-based, will treat absent fields and fields present but with a null value the same. Python, on the other hand, will look for the presence of a key, and according to the logic in json2csv, sometimes it will behave differently depending on whether a key is present (but empty) or absent entirely. This change should obviate this problem, as well as preserve the original structure of the Twitter API JSON.
Updated the JSON schema file so that the full_text
and retweeted_status.full_text
fields are read (if present) by the loader.
Updated branch as follows:
tweetset_loader.py
, line 354) now uses only the subset of fields we want to index.elasticsearch-hadoop
configuration updated to exclude tweet_id
field from indexing.id
. (I was joining on a row number previously, but such numbering is evidently not determinate when loading from multiple files. In other words, loading the same dataset twice can produce different row numberings if the data are distributed across multiple files.) New approach: we load the JSON-L as an RDD, then convert that to a DataFrame, allowing us to preserve the original string representation of the JSON as a separate column and obviating the need for a join (which creates problems on smaller datasets). This approach seems stable, but performance has taken a hit (relative to previous implementations) that is evident on larger datasets. Loading 20 GB with the new implementation (not counting the time to create extracts) took ~35 min, vs. 20 min using the implementation currently in production.
Is that comparison against the previous implementations of Spark 3? Or compared to TweetSets 2.1?
Oh never mind, I see you said compared to production. Sorry!
This versions uses the RDD API for loading to TweetSets (in order to preserve the original JSON as is) and the DataFrame API to create the extracts. Performance is comparable to what's in production for loading and significantly improved for creating extracts.
The Spark SQL code includes fields that we use for indexing in Elasticsearch; these are dropped when creating the CSV. I will leave them there for now (I don't think their presence really impacts performance) with an eye toward a future release where we no longer need to load the full JSON into Elasticsearch. At that point, we can use the DataFrame API for everything (which should improve performance further).
Summary
This branch contains the following changes:
spark-loader
command now uses the Spark DataFrame API to 1) load Tweets into ES and 2) create extracts.Setup
You will need to have an NFS mount shared between your primary and cluster nodes/VM's in order for this branch to work properly. See the instructions in the original issue.
Make sure that
loader.docket-compose.yml
is configured to build the Docker image for theDockerfile-loader
.Likewise for thespark-master
andspark-worker
containers indocker-compose.yml
on both primary and cluster nodes.You'll also need to update your
.env
file to point theDATASET_PATH
variable to the shared NFS mount. (On my VM's, this is/storage/dataset_loading
.)Currently, the data extracts are written to the same directory used for loading datasets. So they will not appear in the TweetSets UI, which is still looking for them elsewhere. But I didn't want to touch
tweetset_server.py
in this branch, given the work Laura has been doing with the Python 3.8 upgrade.Testing
It will be useful to load the same dataset with the regular loader and then with the Spark loader, in order to compare results in the UI.
Dockerfile-spark
.dataset.json
file in the directory with the JSON of the tweets to load.dataset.json
with a name to distinguish it from the previous load.This presumes that your sample dataset is in the
storage/dataset_loading/sample
directory (or whatever NFS mount is mapped to/dataset
in the .ENV file). In my testing, I put the tweet JSON files anddataset.json
in ajson
subdirectory, but that's not strictly necessary.t126-python-38
). c. Remove the server image (on the primary node) to force rebuild:docker image rm ts_server-flaskrun:latest
d. Brink TweetSets back up.Expected Results
Elasticsearch indexing
extended_tweet
andretweeted_status
elements.quoted_status
text fields for tweets of typequote
. (This decision was made in consultation with Laura for the sake of consistency.)Let me know if you see inconsistencies in the indexing that don't make sense with the above.
Dataset extracts
Each should contain one or more zipped files. I've tested the CSV files against those created by
twarc.json2csv
(1.12.1) and documented some minor differences in the data dictionary. Feel free to test them against full extracts created in the UI, but do keep in mind that the current version of TS is using an older version of json2csv.In my testing, Spark created far too many files for the mention extracts. I assume that is a setting that can be configured, but I haven't looked into that yet.
Performance
I am curious to hear how you experience it. I would expect this code to be faster at least for large datasets -- it was so in testing on my own laptop -- but I haven't had a chance to test a large dataset on the VM's. There may be Spark settings we can tweak to improve performance, but my impression is that for some of these, what's optimal depends on the environment, and our dev VM's are not terribly good proxies for production. We might have more success testing this aspect on tweetsets-dev.