[Feature Request] Can some sort of logging be added?

[hkmaly]

Would it be possible to add some sort of logging, that is, for every saved image, store the time, page url, page title, image url and the filename the image was saved as? Not sure where would be easiest to store it in, I suspect just appending a line to file is not something supported by "modern" browsers, so maybe storing in some sqlite file in Local Storage would be easier?

[gyng]

It's possible, but I don't think I'll be implementing this any time soon due to time. What's your use case for this?

[hkmaly]

My use case is that I've added logging into original save-image-in-folder for old firefox (with older format of extensions and no signing) and currently use it in palemoon, but I suspect I might be forced to downgrade to new firefox.

I'm using the extension for reading and saving comics and need the additional information so different script then can go over saved images, match them to the log entries and divide them into chapters, arcs etc. Storing all of that directly into filename would likely go over filesystem limit for it and would definitely be hard to work with ...

Compared to #164, creating new file for every image metadata separately would mean lot of files so I would consider it worse, but I could work with that as well.

[featherice]

I'd really be happy with a way to store urls alongside domain and date somewhere in metadata or some log. Currently i use extension to automatically rename files with all this information, but you can't write proper urls in the filename.

[hkmaly]

I'm still postponing my move to new Firefox due to this ...

@featherice would database in local storage be suitable for you? I mean, quite important part of this ticket, when it will be actually worked on, is where to put the data (and HOW).

(For case I will became desperate enough to try to write it myself, some links: Filewriter, Filehandle)

[featherice]

@hkmaly Anything will do as long as it is in searchable/viewable implementation.

[gyng]

Just popping in to say I'm aware of this, but my job is taking up most of my energy right now and has been for most of the year. If I do get anything done I expect it to be just a simple JSON view stored in local storage. I'm not committing to anything, but maybe I'll get some more free time and emotional capacity to get something done with the year-end holidays.

I'm also taking any PRs for this, but the code is in quite the mess so I'm not really expecting anyone to figure out how to get it done.

[hkmaly]

I'm also taking any PRs for this, but the code is in quite the mess so I'm not really expecting anyone to figure out how to get it done.

Yeah, I heard that and it's one of the reasons I would need to be desperate to try. I suppose if you do some implementation it would then be easier to extend it than to work on it from scratch.

Also, I'm not exactly having much time myself.

[hkmaly]

Any progress on this?

[gyng]

No progress, sorry, haven't made the time to work on save-in for a while.

On Thu, 7 Jul 2022, 10:18 Honza M, @.***> wrote:

Any progress on this?

— Reply to this email directly, view it on GitHub https://github.com/gyng/save-in/issues/159#issuecomment-1176966916, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC2OQHYNZ4ZFBQDWCYR3V3VSY47BANCNFSM4WGZAJKQ . You are receiving this because you commented.Message ID: @.***>

[gyng]

Released a basic implementation in v3.7.0, accessible via the options page. Please re-open if the issue is still not fixed. Extension storefronts might take some time to update.

[hkmaly]

Thank you. I like how you stored everything you could get your hand on, although I probably won't need the favicon. I must admit that at first look I don't see how to get the data outside firefox - it seems it's serialized in some weird way before writing it to sqlite - but it seems firefox is storing everything in this complicated way so I will likely just need to find out how to decode it.

[hkmaly]

Wait. No. Only last entry is saved. Looking in code it's supposed to load the variable, add new entry and store it, but I only see the last entry all the time.

... also, doesn't look like I can reopen.

[hkmaly]

Hmmm ... Chrome also stores it in complicated way.

Got some idea how to handle that without trying to find out better API for storing data: What about adding "Save history" button which would allow to download current history (like, generate normal json from it on fly and open download dialog for it)? That should be relatively simple, wouldn't it?

[hkmaly]

However, good news: Found how to read those firefox database files. Including the howtos I needed

• https://stackoverflow.com/questions/54920939/parsing-fb-puritys-firefox-idb-indexed-database-api-object-data-blob-from-lin
• https://gitlab.com/ntninja/moz-idb-edit
• https://stackoverflow.com/questions/48535799/module-snappy-has-no-attribute-decompress

After installing moz-idb-edit and dependencies - python-snappy was the tricky one, but also jmespath - I also needed to modify reading of dates:

@@ -505,9 +505,12 @@ class Reader:
                        return True, JSBigIntObj(self.read_bigint())

                elif tag == DataType.DATE_OBJECT:
+                       try:
                        # These timestamps are always UTC
-                       return True, datetime.datetime.fromtimestamp(self.input.read_double(),
+                           return True, datetime.datetime.fromtimestamp(self.input.read_double(),
                                                                     datetime.timezone.utc)
+                       except ValueError:
+                           return False, None

                elif tag == DataType.REGEXP_OBJECT:
                        flags = RegExpFlag(data)

because apparently some invalid dates were encountered, but now it works with command line

./moz-idb-edit --extension '{72d92df5-2aa0-4b06-b807-aa21767545cd}' --profile path-to-profile '"save-in-history"'

... the save history button might still be handy, but fixing the issue with only last file being recorded is more important.

[hkmaly]

@gyng Ok, I think I have it. Frankly, most likely the actual error is that it should be current[HISTORY_KEY] instead of current.history, but maybe it would be safer to get rid of those awaits as well.

I tested it like this and it recorded at least three files.

I hope you can merge it soon so I can update and enable xpinstall.signatures.required again. I have no idea how to sign extension.

[hkmaly]

@gyng Wrote alternate implementation, which uses explicit then according to promise documentation instead of callback function which works but apparently isn't documented.

Also, for anyone wanting to get the data outside firefox: I've forked the tool I recommended and will be patching it as necessary (the patch I mentioned here doesn't suffice). See https://github.com/hkmaly/moz-idb-edit

(EDIT: Just few minutes after doing all that work, I noticed that https://gitlab.com/will.foran/moz-idb-edit/ works too.)

[hkmaly]

@gyng Hello? Any progress?

[hkmaly]

@gyng Hello?

[hkmaly]

@gyng Any progress?

[hkmaly]

@gyng Are you still alive?

[hkmaly]

@gyng I see you are still committing to other repositories!