Brainicism / DiscordDataParser

Utility to parse and visualize raw Discord data request information
17 stars 3 forks source link

Not being able to recognize data-path? #51

Closed VirenMohindra closed 6 years ago

VirenMohindra commented 6 years ago

Am I missing something? I cannot analyze the contents of the package after downloading and unzipping the email from Discord, it throws a directory not available when I can definitely cd into it.

screen shot 2018-09-02 at 2 38 46 am

Brainicism commented 6 years ago

Hey @VirenMohindra Not immediately sure what's wrong, but could you try to use a relative path rather than absolute for now? Will look into it in the meantime.

VirenMohindra commented 6 years ago

Moved the package file into the DiscordDataParser-master directory and still no luck. screen shot 2018-09-02 at 3 04 05 am

Brainicism commented 6 years ago

That error is thrown if either:

messages_path = "#{data_path}/messages"
activity_path = "#{data_path}/activity/analytics"

return false when passed into File.directory?, which should return true.. Could you try commenting out activity_analyzer.rb:22 and messages_analyzer.rb:18? Perhaps that check is failing unnecessarily.

Brainicism commented 6 years ago

Found a regression possibly related to this. Working on it now EDIT: Was unrelated

VirenMohindra commented 6 years ago

The discord package I received Sep 1 2018 had the following directory structure:

package
+-- account
|   +-- user.json
+-- activity
|   +-- modeling
      |   +-- events-2018-00000-of-00001.json
|   +-- reporting
      |   +-- events-2017-00000-of-00001.json
      |   +-- events-2018-00000-of-00001.json
|   +-- tns
      |   +-- events-2018-00000-of-00001.json
+-- messages
|   +-- multiple 18 digit integer folder name(s)
      |   +-- channel.json
      |   +-- messages.csv
|   +-- index.json
+-- programs (empty folder)
+-- README.txt
+-- servers
|   +-- multiple 18 digit integer folder name(s)
      |   +-- audit-log.json
      |   +-- guild.json
|   +-- index.json

Note, I've recently started using Discord heavily but I have had an account since Tue Nov 14, 2017.

Not sure if that had anything to do with this issue.

I do not have an analytics folder in the activity folder.

VirenMohindra commented 6 years ago

After commenting out activity_analyzer.rb:22 it seems to add up.

I get this error after running $ ruby app.rb --data-path=package

Begin parsing activity... No such file or directory @ dir_initialize - package/activity/analytics`

Brainicism commented 6 years ago

That's interesting, I just kinda assumed discord has been collecting this data from the beginning. Does the app work fine if you remove @activity_analyzer from app.rb:41?

VirenMohindra commented 6 years ago

Commenting out @activity_analyzer from app.rb:39 and app.rb:41 just divides by 0...

Whats your discord tag? I'd love to work on this project and would like a tighter feedback loop.

I can provide my package folder privately for debugging purposes.

Brainicism commented 6 years ago

That would be appreciated if you don't mind. Feel free to remove most of your messages (since I don't think that's specifically the cause here, maybe leave one or two for sanity). Email is [redacted] . Won't be able to look into it until tomorrow though.

Brainicism commented 6 years ago

Issue is related to the lack of activity data in the backup. My hypothesis is that this setting determines whether or not you have that data. alt text

https://github.com/Brainicism/DiscordDataParser/tree/fix/activity-folder-doesnt-exist-51 addresses this. Will need further work to remove the activity related charts from the output html