Storing archives

[Spootacular]

If archive.is exposes a way to get all archives for a given URL, should the bot fetch and store all of these? Someone could then summon the bot with something like "get archives for " and it could reply with a date-sorted list, for instances like threads about stealth edited articles or "check out this nuked thread lel"

[chuggafan]

I do believe it does, as it uses the same API web.archive.org uses

[chuggafan]

I'll think of that, but I did setup for a SQLite version of the bot, if you could help me on that because I am a complete noob at sql that'd be great

[Spootacular]

I've got some done here, it's probably technically already usable but it's not finished or trustworthy yet. Once the database schema is final I can make a quick tool to convert to/from database and flat files.

I do still need to look into redoing how the bot runs through comments to eliminate that entirely, and the post/reply could be replaced with the bot just iterating over the post to find its comment. That would be two tables and files that could be removed then.

[chuggafan]

i'm confused just about https://github.com/Spootacular/Mnemosyne-2.0-1/blob/archive-tracking/Mnemosyne%20Of%20Mine/SQLBotStateTracker.cs#L27 this line, specifically the |datatracker| part, shouldn't that just be a .

[chuggafan]

I made a slight change to a local copy which makes sense, so that we can put out 2 different versions if neccesary

`IBotStateTracker BotState = null;

if SQLite

        BotState = new SQLBotStateTracker();

else

        BotState = new FlatFileBotStateTracker();

endif`

[Spootacular]

As far as I could find, the connection string requires an absolute path and that, plus the two lines above it are one way to provide that. Found it on stackoverflow here

[Spootacular]

Switching from flat file to SQL could be done using config rather than with preprocessor, one build could do either then. It would just need to instantiate the appropriate class at the beginning of the main loop.

[chuggafan]

Yhea, i could do that, also you could use System.Environment.CurrentDirectory + $"{connectionstring}"

[Spootacular]

Apparently that's "non-trivial" for... some reason. I don't see why that wouldn't also work though.

[chuggafan]

LOL, "non-trival", gotta love stackoverflow sometimes, also i just pushed a commit that does the config file, at the end of the config file if you write SQLite "foobar(whatever)"="{boolean value}" It'll take it

[chuggafan]

I'm looking over your unit tests, apparently every last one is failing, any ideas why?

[Spootacular]

At first it was because it wasn't copying over one of SQLite's dlls when it deployed, I had to add it in a DeploymentItem attribute which then let some pass. The others that were failing were because the queries were returning longs when I had assumed it would be an int, making the casts fail.

[chuggafan]

Ahhh, ok, also, the tests fail on x86 based processor archetectures, isn't there an MSIL version of the SQLite DLL that we're using?

[Spootacular]

The tests probably fail because it's only deploying the x64 dll, if a second attribute were used for the x86 dll it should work. Best I can find, even the MSIL-only package still requires the native dll to function.

[chuggafan]

sigh, why does it work that way? oh right, it's c++, sometimes man, sometimes...., I'll have to think of a way to make it pick the right one, it'll probably involve preprocessor commands

[Spootacular]

Including both in separate x86 and x64 folders seems to work, the managed side chooses which one is needed.

[chuggafan]

KK, ETA on when this is ready for deployment?

[Spootacular]

A day or so still. Even with tests passing I don't fully trust it. I want to make sure it won't choke and die spontaneously.

[chuggafan]

KK, i'll start work soon on getting # of times a link has been archived soon, can you help by making the SQL commands for it? I can do the commenting after, as I have no experience in SQL

[Spootacular]

I'll need to know what the end goal is functionality wise to have an idea what needs to be stored.

[chuggafan]

I was thnking "Link URL: How many time's it's been archived" as a thing, I can easily add that to a comment if you can get me a SQL tracking thingy for it

[chuggafan]

To further clarify, I want to know how many times we've archived a specified URL to be able to just give tracking information and stats, that's all

[Spootacular]

So it doesn't need to keep the actual archive urls, only a count? I've been trying to think of a decent way to do that without being slow and messy, since SQLite doesn't understand the concept of arrays. If all it needs is to keep a count, that's a lot more straightforward.

[chuggafan]

Yhea, only a count

[chuggafan]

Added archive count, have to test internally first