search

AnkiHubSoftware / ankihub_addon

1 stars 0 forks source link

Upload logs to datadog instead of s3 #419

Open RisingOrange opened 1 year ago

RisingOrange commented 1 year ago

It would be convenient to have logs in datadog instead having to download them from s3. We could also send logs to datadog during normal add-on operation to get data about how people are using the add-on.

https://docs.datadoghq.com/api/latest/logs/

andrewsanchez commented 1 year ago

We will probably want to make sure that the add-on is only logging things related to ankihub and nothing else. I think that currently the logs include stuff from anki and other add-ons sometimes. Or at least we should filter all logs from modules except those in a whitelist (e.g., if we want to include logs regarding anki's sync server, etc.)

RisingOrange commented 1 year ago

@andrewsanchez Why? Because of privacy / legal reasons or to keep the logs relevant?

About this: https://community.ankihub.net/t/linux-ankihub-giving-popups-with-errors/343/10 I also think that it is the volume of log statements that causes the logging errors for this user. The add-on even logs responses with all their content which can contain hundreds of notes.

Maybe we should truncate response contents when logging them to something like 1000 characters? This would have the advantage that shorter responses are shown in full and longer ones wouldn't clutter the file.

We can also increase the level of the file handler logger, so that debug level logs are not written to the file and adjust the log levels for the log statements across the add-on accordingly.

A large portion of log files is filled with the log statements generated when the add-on is importing notes. This are the log statements for one note. I think we can log these at the debug level so that they don't end up in the log file. At the end of the import the add-on could log the ids of all updated notes at the info level.

DEBUG 2022-08-08 12:05:17,914 sync 68129 123145683660800 Trying to update or create note:
 NoteUpdate(fields=[FieldUpdate(name='Text', order=0, value='<strong>External hemorrhoids</strong>\xa0are treated with {{c1::<u>resection</u>}} when conservative treatment fails.'), FieldUpdate(name='Extra', order=1, value='<img src="paste-16007343112193_1509457489342.jpg">'), FieldUpdate(name='First Aid', order=6, value=''), FieldUpdate(name='Sketchy', order=7, value=''), FieldUpdate(name='Sketchy 2', order=8, value=''), FieldUpdate(name='Sketchy Extra', order=9, value=''), FieldUpdate(name='Picmonic', order=10, value=''), FieldUpdate(name='Pixorize', order=11, value=''), FieldUpdate(name='Physeo', order=12, value=''), FieldUpdate(name='Bootcamp', order=13, value=''), FieldUpdate(name='OME', order=14, value='<div class="ome-widget"><a href="https://onlinemeded.org/spa/gastroenterology/gi-bleed/acquire?ref=anki"><img src="_OME_AnkiFlashcards_Lesson_1.png"/></a></div><br>'), FieldUpdate(name='Additional Resources', order=15, value=''), FieldUpdate(name='One by one', order=16, value='')], ankihub_note_uuid=UUID('9e3c69f3-6c19-4211-a4f0-0d4108fc44c6'), anki_nid=1511225503071, mid=1659130414530, tags=['#AK_Step2_v12::#OME::03_Gastrointestinal::14_GI_Bleeding', '#AK_Step2_v12::#OME::15_General_Surgery::08_Colorectal', '#AK_Step2_v12::#OME_banner::Clinical::03_Gastroenterology::14_GI_Bleed', '#AK_Step2_v12::#Resources_by_rotation::IM::ome::gi::gi_bleed', '#AK_Step2_v12::$AnKingUpdates::$Errata::v11', '#AK_Step2_v12::#Resources_by_rotation::Surgery::ome::gensurg::colorectal', '#AK_Step2_v12::Original_decks::Dorian::surg::ome::gensurg::colorectal', '#AK_Original_Decks::Step_2::Cheesy_Dorian_(M3)', '!AK_UpdateTags::step2PotentialDupes::Forrest::Combine_haveValue', '#AK_Step2_v12::#Subjects::Gastroenterology::07_Rectum::Hemorrhoid', 'AnkiHub_Update::Content', '#AK_Step2_v12::!FLAG_THESE_CARDS::Potential_duplicates', '!AK_UpdateTags::Step2decks::Cheesy-Dorian-(M3)::Surgery::1-OME::1', '#AK_Step2_v12::!Shelf::Surgery'])
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 AnkiHub id of note 1511225503071 changed from 9e3c69f3-6c19-4211-a4f0-0d4108fc44c6 to 9e3c69f3-6c19-4211-a4f0-0d4108fc44c6
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 Tags were changed to ['!AK_UpdateTags::Step2decks::Cheesy-Dorian-(M3)::Surgery::1-OME::1', '#AK_Step2_v12::#Resources_by_rotation::Surgery::ome::gensurg::colorectal', '#AK_Step2_v12::!FLAG_THESE_CARDS::Potential_duplicates', '#AK_Step2_v12::!Shelf::Surgery', '#AK_Step2_v12::Original_decks::Dorian::surg::ome::gensurg::colorectal', 'AnkiHub_Update::Content', '#AK_Step2_v12::#Subjects::Gastroenterology::07_Rectum::Hemorrhoid', '#AK_Step2_v12::#Resources_by_rotation::IM::ome::gi::gi_bleed', '#AK_Step2_v12::#OME_banner::Clinical::03_Gastroenterology::14_GI_Bleed', '#AK_Original_Decks::Step_2::Cheesy_Dorian_(M3)', '!AK_UpdateTags::step2PotentialDupes::Forrest::Combine_haveValue', '#AK_Step2_v12::#OME::03_Gastrointestinal::14_GI_Bleeding', '#AK_Step2_v12::$AnKingUpdates::$Errata::v11', '#AK_Step2_v12::#OME::15_General_Surgery::08_Colorectal'].
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 Field: "Text" was changed from:
<strong>External hemorrhoids</strong> are treated with {{c1::<u>resection</u>}} when conservative treatment fails.
to:
<strong>External hemorrhoids</strong> are treated with {{c1::<u>resection</u>}} when conservative treatment fails.
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 Field: "Extra" was changed from:
<img src="paste-16007343112193_1509457489342.jpg">
to:
<img src="paste-16007343112193_1509457489342.jpg">
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 Field: "Picmonic" was changed from:

to:
DEBUG 2022-08-08 12:05:17,915 sync 68129 123145683660800 Field: "OME" was changed from:
<div class="ome-widget"><a href="https://onlinemeded.org/spa/gastroenterology/gi-bleed/acquire?ref=anki"><img src="_OME_AnkiFlashcards_Lesson_1.png"/></a></div><br>
to:
<div class="ome-widget"><a href="https://onlinemeded.org/spa/gastroenterology/gi-bleed/acquire?ref=anki"><img src="_OME_AnkiFlashcards_Lesson_1.png"/></a></div><br>
DEBUG 2022-08-08 12:05:17,919 sync 68129 123145683660800 Updated note: anki_nid=1511225503071

I think it should be enough to not log (full) response contents for large responses and to reduce the logging while importing to solve the logging errors the user reported because these things generate the vast majority of log messages.

andrewsanchez commented 1 year ago

@andrewsanchez Why? Because of privacy / legal reasons or to keep the logs relevant?

Just so that we are only getting logs that we are actually interested in to make it easier to search in datadog. Also, it's not free to receive/store logs, so we want to avoid sending things we know we aren't interested in.

Maybe we should truncate response contents when logging them to something like 1000 characters? This would have the advantage that shorter responses are shown in full and longer ones wouldn't clutter the file.

That could work. AnkiHub logs all requests and assigns them a unique ID. Maybe it would be better if we just log the response status code, and this UUID so that we can match it up with the related request in AnkiHub. We can also update our endpoints to return a value(s) that the logger pulls out of the response for logging?

We can also increase the level of the file handler logger, so that debug-level logs are not written to the file and adjust the log levels for the log statements across the add-on accordingly.

Or make that adjustable - only write debug-level logs to disk if enabled in add-on config. That would probably be better.

RisingOrange commented 1 year ago

Maybe it would be better if we just log the response status code, and this UUID so that we can match it up with the related request in AnkiHub. We can also update our endpoints to return a value(s) that the logger pulls out of the response for logging?

Logging the request id in the add-on sounds good! AnkiHub does not log all response contents though, right? So we should at least log short response contents in the add-on.

Or make that adjustable - only write debug-level logs to disk if enabled in add-on config.

Should the default level be info? This would probably be better when having it set to debug causes crashes for some users.

I think a quick solution for the user that created the error report (https://community.ankihub.net/t/linux-ankihub-giving-popups-with-errors/343/10) would be to:

That will reduce the amount of logged information a lot and will be probably enough to not cause logging errors. I can add these changes now and the other changes (uploading logs to datadog, logging request ids) can be added later.