search

DocNow / twarc-csv

A plugin for twarc2 for converting tweet JSON into DataFrames and exporting to CSV.
MIT License
31 stars 10 forks source link

Add `--optimized` mode #55

Open igorbrigadir opened 2 years ago

igorbrigadir commented 2 years ago

The current output favors preserving as much information as possible from the original json, but there is some duplication, and a bunch of columns can be removed as they're rarely super useful.

The new --optimized mode will generate CSVs that drop a bunch of columns to save space:

edit_controls.edits_remaining
edit_controls.editable_until
entities.cashtags
entities.hashtags
entities.mentions
withheld.scope
withheld.copyright
author.id
author.entities.description.cashtags
author.entities.description.hashtags
author.entities.description.mentions
author.url
author.withheld.scope
author.withheld.copyright
geo.coordinates.coordinates
geo.coordinates.type
geo.country
geo.full_name
geo.geo.type
matching_rules
__twarc.retrieved_at
__twarc.url
__twarc.version

(exact list to be revised later)

These are the most commonly not present or duplicate ones, where the missing data can be inferred from the columns left over, or with the cashtags, hashtags, mentions, with twitter-text for example.

Should probably fix #36 and #47 first before this.

edsu commented 2 years ago

I'm interested in hearing where the need for this optimization arose. Was it a problem generating the CSV, or reading the generated CSV in another application? It sounds like the latter?

igorbrigadir commented 2 years ago

Just trying to deduplicate columns and remove mostly empty ones, so more can fit into memory, and other tools like great expectations or pandas profiling have an easier time.

edsu commented 2 years ago

Would being able to write to parquet help in situations like that?

igorbrigadir commented 2 years ago

Yep! Definitely i think #29 goes hand in hand with this - I think all of these things are basically the same task for me to do lol