Scraping comments (and threads)

[bianchi-dy]

I'm not a CS major by training so I'm not sure what data structure might suit storing these best or if it might get corrupted by the text being stored in a CSV. Any ideas?

[ssterman]

Sorry for the delay -- if you are still interested in this, some thoughts:

• a csv should be fine! I wouldn't expect text corruption, though you may have to keep an eye out for special characters and ignore images.
  • you probably want 1 row per comment, with columns for all the data you're interested in, including an ID for the parent comment of the thread, which will let you reference the parent's row (looks like you can get that from the "parent thread" button element)
  • an alternative would be json formatting, but I don't think it would be necessary

[bianchi-dy]

I'm actively working on this again! Not sure why the scraping script ao3_get_fanfics.py is slow for me today but my plan is to follow @ssterman's suggestion:

• 1 row per comment
• Interested data: username, date of comment, parent comment id
• So if a fic has 30 individual comments with 10 threads there will be 30 rows (in the comments array) and 10 unique parent comment ids.