search

danny0838 / firefox-scrapbook

ScrapBook X – a legacy Firefox add-on that captures web pages to local device for future retrieval, organization, annotation, and edit.
Mozilla Public License 2.0
323 stars 65 forks source link

Future of Scrapbook X? #85

Closed mseliger closed 6 years ago

mseliger commented 8 years ago

Hello,

Mozilla is going to eliminate xul addons one or two years (if they follow their plans) (see here: https://blog.mozilla.org/addons/2015/08/21/the-future-of-developing-firefox-add-ons/). Is it possible for you Danny to switch Scrapbook (X) to the new api Web Extensions? Or will the project be dead, when xul addons are not more supported by Firefox? Fortunately the data of scrapbook files are not gone, when there is no addon Scrapbook X more because of the possibility to export to an html tree. But it would be very sadly and circumstantial to switch to another information system (ScrapBook X is my main system). I have used ScrapBook and the successors for at least 10 years (and have nearly never lost data). What do you think about the future of Scrapbook X?

Greetings, Maria

danny0838 commented 8 years ago

This is a very difficult question as well as a critical problem.

If Firefox really drops support of xul addons, yes, the current ScrapBook X will no longer work. Fortunately, ScrapBook X would at least be available in older Firefox versions in the near future,

As for the further future, there would be more issues. Will Firefox survive or die? If yes, will new Firefox APIs offer the essential functions that ScrapBook X currently uses? If no, will there be remarkable projects forking Firefox and keep supporting xul addons?

We don't have the answer currently. And we could only take one step at a time.

githmw commented 8 years ago

I'm a new ScrapbookX user. There is another option, mseliger, and that is to use Pale Moon instead of Firefox. Pale Moon used to be a browser that was a Firefox derivative. But their developers, like you, did not like the direction that Firefox was going. So about a year ago, Pale Moon became a true 'stand alone' browser, no longer tied to Firefox. There are some disadvantages to that, especially for websites that so far refuse to recognize Pale Moon, but Pale Moon does not make changes just for change sake. Rather than dropping ScrapbookX (which runs fine on Pale Moon), you may want to look at Pale Moon to replace Firefox as your browser.

Disclaimer: I have no connection with Pale Moon. I am just a satisfied user.

Harry

mseliger commented 8 years ago

Hello Harry,

if Mozilla is dropping the xul support then Palemoon is in the same position of Seamonkey. Either they accept it (and don't support xul extensions anymore) or they have to develop and support their browser and the extensions without the support of Mozilla. For both browser it will be difficult, because the developer teams are small. I personal use a long time Seamonkey instead of Firefox (with the X Sidebar extension and Scrapbook Plus) - but the differences between Seamonkey and Firefox are growing more (you can use Scrapbook X on Seamonkey - there are only lilttle functions that does not work, because of the different sidebar). I am really doubtfully about the future of Seamonkey and other browser who are build on the base of Mozilla. For first, I've have installed the Firefox ESR version, which will be supported one year longer. I've installed the ESR version because I don't need the new functions of Firefox like "Hello" or "Pocket" and the ESR version needs less ressources on my computer (an about 7 years old notebook of Toshiba). If Mozilla does not support xul anymore, I think many user will use another browser, because they use Firefox because of the extensions. Greetings, Maria

githmw commented 8 years ago

Maria:

What a rapid response! Pale Moon is not in the same position as SeaMonkey, Cyberfox, etc. Pale Moon has been diverting from Firefox for several years, finally splitting completely around Firefox 25.

There are lots of threads on the Pale Moon forum on the Firefox decision to stop supporting xul extensions, beginning about last October. In one of the posts, the primary developer of Pale Moon said:

'The common question "Will Pale Moon have to adopt x or y unwanted feature from Firefox" is always a "no" because unlike Waterfox, Cyberfox, etc. we are indeed not at all tied to Mozilla-central anymore. We have our own development and our own base code. Only features that are specifically ported over or re-implemented will end up in Pale Moon. Nothing will be "automatically adopted because Firefox has it". (For some reason a lot of people don't "get" this yet. Pale Moon is truly independent.)'

So add-ons that currently run on Pale Moon will continue to be able to run. Whether developers will continue to develop their add-ons without Firefox is of course a different question. But as long as add-on versions that will run on pre V. 25 Firefox are still around, they can be used with Pale Moon.

I went though this whole hassle a few years ago. At that time I used Opera, which in my opinion was the most flexible browser, with lots of creative features that had been developed. Here in the US, Opera never had much of a following, but in Europe they had a reasonable market share. But as more and more folks gravitated to Chrome, Opera faced a real dilemma. They finally gave up, and created a Chrome clone. The folks who were used to the 'old' Opera were outraged, as are most of the folks who are now posting on the Firefox forums. In the end, Opera lost their most faithful users. I don't know how they are doing now, but they typical reaction I have heard was "if I wanted Chrome I would get Chrome, not a variation.' I fear this is where the Firefox user base will go.

Because I maintain a few websites, I have several browsers on my computer. I still have Opera 12.15, and go back to it occasionally. I am still amazed at what their capabilities were. When I investigate Firefox add-ons that I might like, I am surprised at how many say 'just like the old Opera'.

Harry

mseliger commented 8 years ago

Hello Harry,

I'll give Palemoon a try - the most extensions which I used works with it - only one not (Deskcuts to create Linux .desktop links, which is HTML5 based). Greetings, Maria

danny0838 commented 8 years ago

@githmw We could try making ScrapBook X more compatible to Pale Moon. If you found something of ScrapBook X work wrong on Pale Moon, please give us a feedback. Thank you.

githmw commented 8 years ago

Maria:

I found Deskcut, which says it works, and Deskcuts, that appears to be for Firefox 4 only--and on Version 1. Which was the one you were looking for?

Here are a couple of links you might want to have. Sometimes finding items in Pale Moon's Help section requires a little searching.

https://addons.palemoon.org/extensions/all-extensions/ This is a short list of add-ons written for Pale Moon.

https://addons.palemoon.org/resources/incompatible/ This lists add-ons that are known incompatable, suspect, fixed, or workarounds.

For others, go to the Mozilla site. You may have to look at some of the older versions to find one that works. In general, if exclusively for Firefox 25 and up, it will not work.

Harry

githmw commented 8 years ago

Danny:

Thanks for your offer to help. Other than the printing issue (which is another thread), it all seems to work fine. But if I find something else, I will let you know.

Harry

danny0838 commented 8 years ago

W've got good news.

I contacted with Firefox recently and found that what they have talked about deprecating XUL and XPCOM is political rather than tehnical—that is, XUL/XPCOM will continue to work in future Firefox, but addons based on them are unlikely—or potentially impossibly—get approved and signed by the AMO.

I could somehow understand this policy since XUL/XPCOMs are so integrative that potential inteferences with native Firefox and with other addons are very, very difficult to be detected. This makes confirming the security and nonconflict of an addon a very time-consuming work.

Since Firefox now requires addon to be signed to be installable, technically or politically deprecation seems similar, but there are still different meaning:

  1. Current XUL/XPCOM addons that have already approved and signed will continue to work in future Firefox. Additionally, since minor bug or compatibility code fixes are less difficult to review, they are unlikely to get reject from signing. Therefore, even if Firefox has API or functionality changes in the future, they would still be available as long as the addon developers continue the basic maintainence.
  2. Even if new addons not currently signed will never get a signature in the future, since they are still technically runnable on Firefox, they will be installable and usable on other special versions of Firefox, such as the Developer Edition, which allows the user to turn off signature restriction of addons. Of course this could result a security hazard and are generally not recommended, but at least they will not die.

There is bad news, though. It is confirmed that ScrapBook X does not work correctly on multiprocess Firefox (Electrolysis). Currently multiprocess is configurable, but it is possible that future Firefox are multiprocess only.

It has been already confirmed that ScrapBook X is technically possible to run on multiprocess Firefox. However, due to the fundamental differences between multi- and single-process Firefox, we need a very, very major rework to make ScrapBook X run on multiprocess Firefox. It will be very time consuming, and we don't know whether Firefox AMO will keep approving and signing it.

Though, ScrapBook is so old that there are so many fundamental structure issues that makes optimization and some features difficult to implement. A total rework—let's call it "ScrapBook X 2.x"—is in planned, which will not only make ScrapBook X runnable on multiprocess Firefox but solve many fundamental issues.

An important thing to notice is that, to solve the fundamental issues means a fundamental data structure change, which means that data collected via current ScrapBook X will not be fully compatible in the future version and vice versa. We will provide a conversion tool from 1.x to 2.x, but there could be issues and side effects, though. Also notice that conversion from 2.x to 1.x may be impossible or unsupported.

We will still use XUL/XPCOM technique since addon SDK is Firefox-only (i.e. not available on SeaMonkey and Pale Moon, etc) and WebExtension is incomplete, and, although not well confirmed, they are likely to have a functionality limitation and cannot fit all technical requirements for ScrapBook X.

Support for Firefox Mobile is still pending. What we already know is that Firefox Mobile doesn't use XUL, but could use bootstrap. The difference between XUL/XPCOM and bootstrap is less, and we'll try to make it work on Firefox Mobile as long as it's possible, but it's lower prior, though.

Here is a brief list about the planned features of ScrapBook X 2.0:

Unfortunately, I am personally becoming busier and busier, and when will these plans come true is unbeknown.

mseliger commented 8 years ago

Hello Danny,

thank you for your reply and for developing ScrapBook. I hope that multiprocessing firefox will not come to soon.

Greetings, Maria

githmw commented 8 years ago

Danny:

Thanks for the update and complete explanation of the problems. This sounds like a huge amount of work for anyone other than the largest add-on developers. I think Firefox/Mozilla is going to lose the developer base that helped make them so popular.

I am very thankful that Pale Moon is not going in the direction that Firefox is.

Harry

ferdnyc commented 8 years ago

@danny0838 wrote:

Since Firefox now requires addon to be signed to be installable, technically or politically deprecation seems similar, but there are still different meaning:

  1. Current XUL/XPCOM addons that have already approved and signed will continue to work in future Firefox. Additionally, since minor bug or compatibility code fixes are less difficult to review, they are unlikely to get reject from signing. Therefore, even if Firefox has API or functionality changes in the future, they would still be available as long as the addon developers continue the basic maintainence.
  2. Even if new addons not currently signed will never get a signature in the future, since they are still technically runnable on Firefox, they will be installable and usable on other special versions of Firefox, such as the Developer Edition, which allows the user to turn off signature restriction of addons. Of course this could result a security hazard and are generally not recommended, but at least they will not die.

There's also potential for a little more relief there — as part of the move to enforced signing requirements, it seems Mozilla will/do provide an API allowing unlisted add-ons (ones that don't appear in the http://addons.mozilla.org/ database) to be put through a totally-automated signing process. (See the FAQ at the Extension Signing wiki page starting at "How do I get my add-ons signed if they are not hosted on addons.mozilla.org (AMO)?", and the updated developer Review Policies for details on unlisted add-ons.)

It's possible they'd start rejecting all XUL add-ons for automated or quick-reviewed signing, even unlisted ones that they're not distributing, once the technology is deprecated for extension building. But given the amount of pushback they'd get from the free-software community, especially on Linux, I think they'd be in a world of hurt if they went down that road.

(The community has already been up in arms enough about the move to non-overridable signature enforcement, not-incorrectly comparing it to an enforced DRM scheme—regardless of Mozilla's motivations for imposing it—and questioning whether distributions which maintain a policy of including only unencumbered, non-proprietary free software can continue to distribute mainline Firefox once it can no longer be made to accept unsigned addons. Hence the push from FFx46 to FFx47 for the removal of that switch, as they kick the can while trying to figure out how to avoid alienating entire user communities.)

So, while it wouldn't be ideal, it's possible that a signed version of Scrapbook X could still be distributed directly (possibly even via Github), and still be functional in official release builds of Firefox even if they start rejecting it for listing and distribution on the official AMO site.

danny0838 commented 8 years ago

@ferdnyc Thank you for the information. We recently got a rather bad news: Firefox would shift from Gecko to Servo, which at least means that XUL will be TECHNICALLY unavailable, and potentially XPCOM will also be unavailable. Maybe it wouldn't come up too soon, but it's still a strike for developers who attempts to stick to XUL/XPCOM.

I recently tried developing using Addon-SDK and WebExtension, and it seems that they don't have enough functionality for ScrapBook without using XPCOM, and it's quite difficult to build a UI similar to current without XUL. Before Firefox extend them the only viable way is to stick to XUL and XPCOM.

githmw commented 8 years ago

Danny:

So sorry to read your latest post about Servo. Firefox seems to be determined to eliminate the folks who create the unique add-ons that made Firefox so popular to begin with.

Harry

ertyz commented 7 years ago

Scrap-page for evernote have enough functionality to save and process the page. So never said never

bytos commented 7 years ago

ScrapBook X is my only reason for using Firefox.

Firefox 57 will only run WebExtensions, a good reason to move to Vivaldi.

Probably keeping a ESR copy just to use ScrapBook X when needed.

danny0838 commented 7 years ago

Here's a rather good news: we just proved that it's basically possible to save a web page with current available WebExtension API. Here are the demos: Firefox version (use Firefox Developer Edition since it's not reviewed), and Chrome version.

However, there are still many many limitations in WebExtension, such as:

  1. For Firefox, saving a web page or a file from file:/// is not allowed.
  2. For Chrome, the user must disable "Ask where to save each file before downloading.", or he will get numerous prompts on each saving.
  3. Local file access is very very restricted, and therefore editing, saving, indexing, and fulltext searching saved web pages is probably not possible.
yfdyh000 commented 7 years ago
  1. Wish Bug 1266960 - Extensions can not load file: URLs
  2. @danny0838 Use jsZip for downloaded (Cached) files to save an zip will helpful?
  3. Wish Bug 1246236 - Implement local filesystem read/write access.
  4. Forefox -> Firefox.
ferdnyc commented 7 years ago

As an alternative on number 3 above (the 1246236 bug that @yfdyh000 mentioned), there's a related bug 1331618 to provide extensions with an optional permission allowing them to store unlimited client-side data in an indexedDB without requiring any sort of confirmation popups. While I would prefer that Scrapbook continue to store its content in a raw filesystem tree (and in fact my own Scrapbook use currently relies on it), it's probably worth at least investigating the possibility of using an indexedDB for the saved content. The "Export to HTML" process could do the job of writing the indexedDB's store out to the filesystem when needed, or if that proves unworkable (and assuming the indexedDB itself isn't encrypted) perhaps an external script.

danny0838 commented 7 years ago

@ferdnyc Technically we do can store much data in the indexedDB, but the problem is: how can the user view a web page and resources saved in the indexDB?

One approach is: when the user tries to open a page which is stored in the indexDB, trigger a series of downloading processes and then open the index.html afterwards automatically. This is probably the safest way, but the performance would probably be hampered very much. (And Chrome users will face the prompting issue as mentioned above.)

Another approach is to serve those pages and linked resources via data or blob URI using a complicated conversion technique (since they don't support relative links, all embeded resources have to be served via data or blob URI as well). The problem is that blob URI may face a session change issue (if a pages links into another page, previous blobs in the page will be no longer available and need to be generated again), and data URI may face a serious performance degradation or even be unavailable for large files, such as a video in gigabytes. Also, data or blob URIs cannot be manipulated by WebExtension since content script is only available for http(s) and file protocols.

The third possible approach is to serve those pages in an iframe of an extension page with content script to manage it. The issue is that the webpage could have its own script that refuses the page to be served in an (i)frame.

Is there any additional possible approach? I haven't come up with one yet...

External script could be a solution. But if we have to use it (and to install another application on the computer or possibly mobile), why don't we just let the user do the management job using whatever external application he wants?

ferdnyc commented 7 years ago

Interesting, thanks for the explanation. My suggestion was definitely based on the asumption that Scrapbook would be able to store its content only in the indexedDB, and allow the user to browse it directly from there using the sidebar, under normal circumstances. The "Export to HTML" / external-script process for writing it out to the filesystem would (in my thinking) only be necessary for the relatively few users who actually require disk files for some other reason. (In my case, because I rsync the Scrapbook directory to my external-facing server so that I can access it remotely via the web.)

The discussion/argument on... one of the Mozilla bugs regarding extension file:/// access (can't recall if it was one that @yfdyh000 posted, my own, or a different one entirely) was admittedly far-ranging and at times heated (to the point that Mozilla moderators twice called on those involved to take it to a mailing list, rather than further clutter the bug), but the proponents of the "no file:/// access for extensions" position made it sound fairly doable to use the indexedDB as a replacement for anything that might otherwise be stored on disk. Certainly you'd think that for web content, it would be, since Firefox is still theoretically a browser. Sort of falls into the "You had ONE job..." category.

But I definitely haven't tried it myself, and was only going by my read of what was said (by one side) in that argument. (Aha, it was this argument: Bug 1246236 - Implement local filesystem read/write access, which ­— fair warning — extends to 86 comments, many of them quite long, and goes around in circles more than a couple of times.)

danny0838 commented 7 years ago

Updated the version as well as the download links: Firefox (signed), Chrome.

Also created the repository. You can add your feedback there.

eliotb commented 6 years ago

Just an uneducated thought. Zotero recently switched from being fully in-browser to being a standalone app + browser helper. AFAIK the standalone app still basically uses mozilla libraries i.e. stripped down version of firefox.

Could something similar work for scrapbook.

danny0838 commented 6 years ago

Close this issue as our policy is clear now: We won't add support of e10s and WebExtension for ScrapBook X. For a WebExtension "port", use Web ScrapBook.

We'll keep basic maintenance for ScrapBook X, but new features and large code rework are unlikely.