search

strohne / Facepager

Facepager was made for fetching public available data from YouTube, Twitter and other websites on the basis of APIs and webscraping.
https://github.com/strohne/Facepager/releases
506 stars 198 forks source link

Can't export note data because it's too large #106

Closed dhsdshdhk closed 5 years ago

dhsdshdhk commented 5 years ago

I collected about 700k twitter posts to a node and I can't exhibit in facepager or export them because it fills up my 16G of RAM and freezes my computer. I then made a 16G swap partition, it was filled up to 10G but it was just hanging forever, so I quit the program. I know this is my problem for having too little RAM, but do you know a way I could successfully export this data to csv?

strohne commented 5 years ago

I suggest you try reading the database file with a database program and prepare the data with a statistics program of your choice. See the wiki for an example of reading the database file with R: https://github.com/strohne/Facepager/wiki/Data-Analysis#reading-the-database-file

To see the content of the file you can use https://sqlitebrowser.org/