search

Tyrrrz / DiscordChatExporter

Exports Discord chat logs to a file
MIT License
7.66k stars 700 forks source link

System.InvalidOperationException: Cannot read the specified JSON element as a non-empty and non-whitespace string value. #767

Closed ivan closed 2 years ago

ivan commented 2 years ago

Version

65ab003ff56511f70096aef6487e0d683feaee92

Flavor

CLI (Command Line Interface)

Export format

JSON

Details

I observed this while running the CI build for 65ab003ff56511f70096aef6487e0d683feaee92 on NixOS:

$ mkdir logs && dotnet DiscordChatExporter.Cli.dll exportall -t TOKEN -f Json --parallel 8 -o logs
[...]
Need Help / help-react                        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  72%
Need Help / help-threads-react                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Need Help / help-react-native                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  99%
Need Help / help-js                           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Need Help / help-styling                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Need Help / help-backend                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Need Help / help-graphql                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Need Help / code-review                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / general-react                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  53%
React General / testing                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / tooling                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / react-internals               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / internationalization          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / accessibility                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
React General / job-board                     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Q&A / q-and-a                                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / tech-reads-and-news               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / events                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / jobs-advice                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / i-built-this                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / i-wrote-this                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / content-creators                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / general-tech                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / functional-programming            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / ux-ui-design                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Community / random                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  95%
Hobbies / gaming                              ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / finance                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0%
Hobbies / fitness                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / music                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / food                                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / movies-and-tv                       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / sports                              ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / photography                         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / bikes-cars-outdoors                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Hobbies / and-more                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / react-router                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  47%
Libraries / redux                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  36%
Libraries / redux-saga                        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / react-bootstrap                   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / formik                            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0%
Libraries / fbt                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / rx-js                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / refract                           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / atomic-layout                     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Libraries / react-beautiful-dnd               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
Tools / webpack                               ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  38%
Tools / gatsby                                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  46%

ERROR
System.InvalidOperationException: Cannot read the specified JSON element as a non-empty and non-whitespace string value.
  at JsonExtensions.Reading.StringExtensions.GetNonWhiteSpaceString(JsonElement element) in /_/JsonExtensions/Reading/StringExtensions.cs:55
  at DiscordChatExporter.Core.Discord.Data.User.Parse(JsonElement json) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Discord\Data\User.cs:38
  at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
  at DiscordChatExporter.Core.Discord.Data.Message.Parse(JsonElement json)
  at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
  at System.Linq.Enumerable.ReverseIterator`1.ToArray()
  at DiscordChatExporter.Core.Discord.DiscordClient.GetMessagesAsync(Snowflake channelId, Nullable`1 after, Nullable`1 before, IProgress`1 progress, CancellationToken cancellationToken)+MoveNext()
  at DiscordChatExporter.Core.Discord.DiscordClient.GetMessagesAsync(Snowflake channelId, Nullable`1 after, Nullable`1 before, IProgress`1 progress, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:43
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:43
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:78
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_2.<<ExecuteAsync>b__2>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:102
  at DiscordChatExporter.Cli.Utils.Extensions.ConsoleExtensions.StartTaskAsync(ProgressContext progressContext, String description, Func`2 performOperationAsync) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Utils\Extensions\ConsoleExtensions.cs:49
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_1.<<ExecuteAsync>b__1>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:109
  at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_0.<<ExecuteAsync>b__0>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:70
  at Spectre.Console.Progress.<>c__DisplayClass27_0.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:103
  at Spectre.Console.Progress.<>c__DisplayClass28_0`1.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:138
  at Spectre.Console.Internal.DefaultExclusivityMode.Run[T](Func`1 func) in /_/src/Spectre.Console/Internal/DefaultExclusivityMode.cs:44
  at Spectre.Console.Progress.StartAsync[T](Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:121
  at Spectre.Console.Progress.StartAsync(Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:101
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExecuteAsync(IConsole console, IReadOnlyList`1 channels) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:68
  at DiscordChatExporter.Cli.Commands.ExportAllCommand.ExecuteAsync(IConsole console) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\ExportAllCommand.cs:38
  at CliFx.CliApplication.RunAsync(ApplicationSchema applicationSchema, CommandInput commandInput) in /home/runner/work/CliFx/CliFx/CliFx/CliApplication.cs:147
  at CliFx.CliApplication.RunAsync(IReadOnlyList`1 commandLineArguments, IReadOnlyDictionary`2 environmentVariables) in /home/runner/work/CliFx/CliFx/CliFx/CliApplication.cs:191

That points to https://github.com/Tyrrrz/DiscordChatExporter/blob/e31ff55e33605906ea375ed90f7ae30fadb412fc/DiscordChatExporter.Core/Discord/Data/User.cs#L38

This can probably be reproduced by joining https://discord.com/invite/reactiflux and https://discord.com/invite/gamedev

Steps to reproduce

  1. Join https://discord.com/invite/reactiflux and https://discord.com/invite/gamedev
  2. With CI build, mkdir logs && dotnet DiscordChatExporter.Cli.dll exportall -t TOKEN -f Json --parallel 8 -o logs
Tyrrrz commented 2 years ago

Hi! Can you please isolate this to a specific channel where this can be reproduced? So that I don't have to run exportall.

ivan commented 2 years ago

Sure, I am running things now to figure out which of the 8 channels it was. I should have something soon.

Tyrrrz commented 2 years ago

Sure, I am running things now to figure out which of the 8 channels it was. I should have something soon.

Thanks!

ivan commented 2 years ago

Thanks for the ping, I forgot this finished :)

The issue is with channel 85338836384628736 in https://discord.com/invite/gamedev -

# dotnet DiscordChatExporter.Cli.dll export -t TOKEN --channel 85338836384628736 -f Json -o gamedev-chat
Exporting 1 channel(s)...

Chat / gamedev-chat ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  74%

ERROR
System.InvalidOperationException: Cannot read the specified JSON element as a non-empty and non-whitespace string value.
  at JsonExtensions.Reading.StringExtensions.GetNonWhiteSpaceString(JsonElement element) in /_/JsonExtensions/Reading/StringExtensions.cs:55
  at DiscordChatExporter.Core.Discord.Data.User.Parse(JsonElement json) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Discord\Data\User.cs:38
  at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
  at DiscordChatExporter.Core.Discord.Data.Message.Parse(JsonElement json)
  at System.Linq.Enumerable.SelectEnumerableIterator`2.ToArray()
  at System.Linq.Enumerable.ReverseIterator`1.ToArray()
  at DiscordChatExporter.Core.Discord.DiscordClient.GetMessagesAsync(Snowflake channelId, Nullable`1 after, Nullable`1 before, IProgress`1 progress, CancellationToken cancellationToken)+MoveNext()
  at DiscordChatExporter.Core.Discord.DiscordClient.GetMessagesAsync(Snowflake channelId, Nullable`1 after, Nullable`1 before, IProgress`1 progress, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:43
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:43
  at DiscordChatExporter.Core.Exporting.ChannelExporter.ExportChannelAsync(ExportRequest request, IProgress`1 progress, CancellationToken cancellationToken) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Core\Exporting\ChannelExporter.cs:78
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_2.<<ExecuteAsync>b__2>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:101
  at DiscordChatExporter.Cli.Utils.Extensions.ConsoleExtensions.StartTaskAsync(ProgressContext progressContext, String description, Func`2 performOperationAsync) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Utils\Extensions\ConsoleExtensions.cs:43
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_1.<<ExecuteAsync>b__1>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:81
  at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.<>c__DisplayClass43_0.<<ExecuteAsync>b__0>d.MoveNext() in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:70
  at Spectre.Console.Progress.<>c__DisplayClass27_0.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:103
  at Spectre.Console.Progress.<>c__DisplayClass28_0`1.<<StartAsync>b__0>d.MoveNext() in /_/src/Spectre.Console/Live/Progress/Progress.cs:138
  at Spectre.Console.Internal.DefaultExclusivityMode.Run[T](Func`1 func) in /_/src/Spectre.Console/Internal/DefaultExclusivityMode.cs:44
  at Spectre.Console.Progress.StartAsync[T](Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:121
  at Spectre.Console.Progress.StartAsync(Func`2 action) in /_/src/Spectre.Console/Live/Progress/Progress.cs:101
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExecuteAsync(IConsole console, IReadOnlyList`1 channels) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:68
  at DiscordChatExporter.Cli.Commands.Base.ExportCommandBase.ExecuteAsync(IConsole console, IReadOnlyList`1 channelIds) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\Base\ExportCommandBase.cs:163
  at DiscordChatExporter.Cli.Commands.ExportChannelsCommand.ExecuteAsync(IConsole console) in /D:\a\DiscordChatExporter\DiscordChatExporter\DiscordChatExporter.Cli\Commands\ExportChannelsCommand.cs:19
  at CliFx.CliApplication.RunAsync(ApplicationSchema applicationSchema, CommandInput commandInput) in /home/runner/work/CliFx/CliFx/CliFx/CliApplication.cs:147
  at CliFx.CliApplication.RunAsync(IReadOnlyList`1 commandLineArguments, IReadOnlyDictionary`2 environmentVariables) in /home/runner/work/CliFx/CliFx/CliFx/CliApplication.cs:191

The last part saved in the JSON file is:

    {
      "id": "710628432295362631",
      "type": "Default",
      "timestamp": "2020-05-14T23:03:41.097+00:00",
      "timestampEdited": null,
      "callEndedTimestamp": null,
      "isPinned": false,
      "content": "I am noob to unity",
      "author": {
        "id": "591450023695351808",
        "name": "YABOY",
        "discriminator": "8234",
        "nickname": "YABOY",
        "color": null,
        "isBot": false,
        "avatarUrl": "https://cdn.discordapp.com/avatars/591450023695351808/f44762884d742cab6c2c408e559c4139.png?size=128"
      },
      "attachments": [],
      "embeds": [],
      "reactions": [],
      "mentions": []
    }
  ],
  "messageCount": 2185700
}
ivan commented 2 years ago

Reproduces quickly with

dotnet DiscordChatExporter.Cli.dll export -t TOKEN --channel 85338836384628736 -f Json -o gamedev-chat --after 710628432295362631
Tyrrrz commented 2 years ago

Thanks, reproduced. Apparently this message, containing a mention to a user with seemingly no username, breaks it: image

Tyrrrz commented 2 years ago

Please try the latest CI build

ramoneres commented 2 years ago

Getting the same error on the GUI version. Same issue, a message that mentions a user with no username throws an exception. Did you only fix the CLI version?

96-LB commented 2 years ago

@ramoneres There hasn't been a new stable version released since this bug was fixed. You can download one of the CI builds here if you want to get a version of the program after any particular commit.

Edit: a new version was just released.