Is there really no way for SF to open a file from the filesystem or read its content?

[user0022]

Hello!

Browser: Firefox. OS: Lubuntu 18.04.

Browser limitation problem: SF can not access filsystem? Recently I read here in the SF issues the developer saying "it's technically impossible for an extension to read the content of a file from the filesystem". This as a negative answer and explanation to the feature request "Indicate that a page has already been saved". Or "it's impossible, from a technical point of view, to open the saved file in a tab. Extensions can't open tabs with file:/// URIs.", answered in "(...) close the (...) tab, then open the file saved by singlefile".

Features it could permit maybe. I was disapointed for I had been hoping certain features in this addon that finally relied on this access that seems to be impossible. Like the 2 just mentionned:

• Indicate that a link or a tab has already been saved (with a color or a mark somewhere?).
• Close and auto-save a page (or save and auto-close, both already existing), and then replace it by its saved version. That would avoid of course a repetitive task.
  • That might also fix a tab suspending addons flaw, for as far as I remember, they sometimes (or always?) empty the page and it must be taken again from the web, a thing that someone who saves his pages might want to avoid. Even Firefox default suspending seems to malfunction as when I restart it, not focused tabs are suspended and should show their content on click, but every other time they don't, even with much free space in cache. Now reopening a suspended tab would always work if the page is in the hard saved version.
  • That could also lead to replace the cache, with some advantages, no size limit, and never erasing!
  • And maybe also if possible, then empty the cache of the "live page", to lighten it, as this version is more or less useless now.

And other ones, like:

• Automatically prevent to save an already saved page (but probably a possibility to force it). Useful or indispensable, especially for "batch save" or auto-save.
• Have ability to check a "last modification date" that must be included in the web page code, and to compare it to the one in the saved page. Pages utilized elements like this one could be indexed in a file maybe, to be faster. That could make possible to:
  • Update a saved page, e. g. if new comments.
  • When opening a URL, open the saved version by default, and the "live page" only if it's newer. For opening by default the saved page would be great I guess, but without this comparison possibility, it could keep the user stuck to old pages.
  • This could also fix a small problem that I think that exists with the SF "Bookmarks" option, when you check link the new bookmark to the saved page [x]. Because then the bookmark white star icon in Firefox address bar, which turns blue to say "already bookmarked", now does it for the saved page URI, but not anymore for the "live" web page. That seems logical as it is the URI that has been bookmarked, but maybe a user would have liked to keep both. But with opening by default the saved page URI when opening a URL, this feature that links a bookmark to the URI seems not needed anymore, so if suppressed, its flaw is suppressed with it.
• Etc?

Solutions I tried to find (Maybe it's the 4/ that would work). It's too hard for me to believe that an addon can't perform such elementary tasks, like to know the addresses of the pages it has itself saved! So I'd just like to try suggestions, well as I'm zero-level, excuse-me if ideas are too stupid. But who knows. I just searched where or how the URI or the content of a file (URL or last modification date or else) could be accessible to SF.

1/ To access page content/elements: in Firefox "itself"? I just had the idea (??) that a place in the Firefox "body" (interface?) could be reachable by SF to write and read the URLs or else.

• In the content of an open tab? A "dummy" or "virtual" tab maybe, that could be invisible if possible. If the list (of e. g. URLs of already saved pages) is somehow displayed in a FF open page, as if it were a web page, or an addon options page, then it is accessible?
• In a SF menu item?
• With the Downloads folder open in a tab? As Firefox can display local folders and files, simply for instance with /home/[myself]/Downloads/ in address bar. Maybe once URIs displayed there, SF can read the URLs in the file of each URI or open the local file which would content their list?
• ... (The last 2 ones a bit too imaginative I think but I tell them anyway, in case it was the only accessible place!)
• In the address bar of a tab? ("Virtual tab" too?) I think that an addon can read this place? So maybe instead of having there one URL, there could be a big list. I tested it by copying a 1Mo text in it and it took it. Same test with 5Mo was a bit too much for my old PC. But even if possible, I imagine conflicts with some Firefox functions, and with many addons that would use the address bar text. Maybe also an address bar could be isolated or reserved to SF but well.
• In the SF icon? Maybe like a text hidden in an image, made invisible but nevertheless accessible.

2/ To access page content/elements: in a session manager or blacklist addon way? Addons that save browsing sessions, if I understand, they success in saving URLs somewhere "that can be read"? (like MySessions, which choose to make it in Bookmarks but others didn't). But I feel that I'm torally wrong. And maybe also, displaying them is a thing, but comparing them and triggering an action after that is something else. Maybe addons that can cleanup Bookmarks by e.g. removing duplicates can help? A bit the same thing with the blacklist addons. And probably other kinds of addons which can save and read URLs. So it it possible to copy their functionning?

3/ To access page content/elements or for URIs opening: remotely?

• When saving pages, SF would send their URLs (or other elements) in "a distant server" or "a web service" or "the cloud" (I don't know what and what name). And before saving a page or not, SF would compare the URL to this list.
• SF could perhaps communicate with a remote software which him can see what is on "local disk", and can afterwards send to local files opening actions?

4/ To access page content/elements or for URIs opening: with an additional software? Maybe an optional additional script or software (to download separately?), could read the saved pages, and communicate with SF bypassing Firefox? Or e. g. detect a page has been saved and closed by SF and open its saved version. In this case I have the hypothesis (??) that despite an addon can not read the user's "local content", an installed software can do the inverse, that is talk to the addon. If SF is still prevented to communicate with this software because it is installed "locally", maybe the communication between them can be done remotely? (web etc.). Such a functioning could avoid to store lists like in previous proposals, the software could just send a signal "already saved" or "same last modification date". I just saw the existence of the "Companion" program. I didn't try it, but could it be the "external software" I thought of? Later addition: an addon existing that works this way?. I came accros the Local Filesystem Links addon, which seems to have made this, adding a software to access the local files. I have the impression that maybe if this access to filesystem is implemented, it will be like this addon.

Conclusion. Conclusion is......... suspense.... 0! Nothing works thank you. Go to bed now. :/

[user0022]

Not sure if this too long post was worth it. But I edited it to make it clearer, and I added a link to "Local Filesystem Links" addon in the end, that seems to do what I was talking about!

[gildas-lormeau]

Thank you @user0022, I promise I'll read it conscientiously soon ;)

[unphased]

So, first off, I think the Companion would certainly be able to read the files that it's responsible for saving, being a node script.

I think that means that if Companion-based saving is enabled, then this definitely would be capable of providing the feature of not auto-saving indiscriminately, and as an example preserve only the most recent version (or n-most recent versions) upon re-navigating to the same page. I'll probably implement that at some point so I'll be sure to make a PR for that.

I haven't tested Companion yet, but I've read the description of how it works and I'm looking at the code now... It looks like it's possible to make it do what I want, which would be to get the savepage content generated by the extension running on the actually-being-used browser, sent to the Companion, which will just shove the content into the target file on disk.

To me, the companion is clearly the way to go because it's rather absurd to have to deal with an endless stream of SingleFile page archives in your browser's download history and download bar. I'm not going to just disable the download bar because I want the regular experience of having that bar when a normal file is downloaded.

I'm not sure if it's possible to get companion set up without requiring any browser backend dependencies, but I'll figure that out.

Sorry for hijacking your issue, although my particular concerns do seem related enough to your questions. Now I think that the externalSave approach from the linked code is the behavior that I was worried it would use, it is the less-than-ideal one, where a headless backend is used to fetch the content. That's fine in theory for most sites, but all content that would require cookies or various login state (or whatever other context) to actually load, they wouldn't work in the context of a private headless browser backend instance.

I'll pick this apart more soon once I get into spinning up this automation.

Huge kudos to @gildas-lormeau for this tool and also already having implemented like 95% of the features that I wanted to build for myself. This is going to be a huge game changer.