A Large Substitution List Disables FireFox

[GoogleCodeExporter]

If there are many regex rules FF freezes until FoxReplace completes. Not just 
the window where the tab being operated on is, but every FF window. I have more 
than 1Gig free RAM during the process. Is this a problem with JAVA or with FF 
or with FoxReplace?

What version of the product are you using? On what operating system?

FireFox 20.0.1
FoxReplace 0.15.0
Windows 7

Original issue reported on code.google.com by sqgl2...@gmail.com on 26 Sep 2013 at 4:30

[GoogleCodeExporter]

It has nothing to do with Java because Java isn't used. Also, I don't think it 
has anything to do with RAM unless the Firefox process is consuming 1 GB or 
more. So, I suppose the freeze is due to a high CPU usage for a while.

Maybe there is some part of the code that can be optimized, but I can't do 
anything without a specific reproducible case. Can you provide a substitution 
list and a webpage where this is reproducible? Also, how much freeze time are 
we talking about? Less than a second? A few seconds? More? And what CPU do you 
have?

Original comment by marc.r...@gmail.com on 26 Sep 2013 at 8:33

[GoogleCodeExporter]

Overall CPU's is about 12% and none of them is maxing out. Every second CPU is 
idle (to be expected since each pair is not two physically discrete CPU's 
AFAIK).

Freeze lasts minutes. But the substitution list is probably the longest one you 
have seen :)

i7-2630QM @2GHz
Windows7 in 64 bit mode

The pages I am operating on are password protected so I cannot give access to 
them easily but maybe you can edit the json subtitutions file so that it 
applies to *any* random page (aferall actual successful substitutions are not 
necessary to put it through its paces, just checking for matches should be 
intensive enough). There are *many* subtitution lists so manually editing the 
names of the pages they all apply to would be tedious so hopefully you can do a 
global replace in a json editor.

Original comment by sqgl2...@gmail.com on 26 Sep 2013 at 9:21

Attachments:

• FoxReplace_130926a.zip

[GoogleCodeExporter]

Hi, I haven't been able to reproduce a long freeze, even replacing the urls by 
*, but I have some guesses.

First: I have observed that most substitution groups use HTML. Every time the 
HTML is changed Firefox must render the whole page again. The HTML is changed 
at most one time in a substitution group (the HTML is copied from innerHTML, 
all substitutions applied, and the result is assigned back to innerHTML), but 
there are a lot of groups sharing the same URL, so the page may be re-rendered 
several times. This could be improved from my part by applying all the HTML 
substitutions from all the matching gropus before assigning the result to 
innerHTML.

Second: For each individual substitution the string is always replaced. I don't 
know how the JavaScript engine handles this when the RegExp doesn't match and 
the result is the same as the input, i.e. I don't know whether it creates a 
deep copy or a shallow copy. I could try to first check if there is a match and 
then apply the substitution only if it matches. I'd have to try if this is 
faster or slower.

I guess that the combination of both factors (or just the first) and the large 
number of substitution groups and substitutions in each group is the cause of 
the problem, and maybe also the "complexity" of the affected webpages.

Original comment by marc.r...@gmail.com on 26 Sep 2013 at 5:05

[GoogleCodeExporter]

Original comment from 
https://code.google.com/p/foxreplace/issues/detail?id=130#c2

#2 google_junk@jo.koan.net

The situation is worse than this.

Large pages (or long substitution lists) can make FoxReplace either 
(1) complete but not with all substitutions executed
(2) hang indefinitely (while hogging resources) or
(3) crashing

To be fair, I am talking about a *huge* file and/or a *huge* substitution list. 
You can test the limits via the files at http://jo.koan.net/foxreplace (file 
names and explanation in the screenshot png should be enough but let me know if 
it does not make sense).

Am using FireFox 26.0

Original comment by marc.r...@gmail.com on 26 Dec 2013 at 9:15

[GoogleCodeExporter]

Marc although you already have two workarounds in mind to remedy the problem 
may I suggest you apply them to a *paid* version? It really takes large/complex 
pages/substitutions to encounter the problem and I would be happy to pay $20. I 
am not using it professionally but imagine most people encountering this would 
be professionals so they ought to contribute to your efforts.

Original comment by sqgl2...@gmail.com on 29 Jan 2014 at 12:20

[GoogleCodeExporter]

Thank you for your suggestion of a paid version, but it's not in my plans. I 
wouldn't like to have a slower free version and the full-speed paid one, 
neither from a philosophical point of view or from a practical one (having to 
maintain both implementations). Also, the user base isn't big enough to make 
any significant money out of it, considering only a small percentage of users 
would need the paid version.

I haven't had much time lately to dedicate to FoxReplace development, but 
seeing that you really need this, I will step up the priority on this issue so 
it comes at the beginning of the queue when I resume.

Original comment by marc.r...@gmail.com on 29 Jan 2014 at 7:17

• Changed state: Accepted
• Added labels: Performance, Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Thanks for escalating the priority. It may even be related to other issues 
people have logged(?)

marc sez...
> Also, I don't think it has anything to do with RAM unless the Firefox process 
is consuming 1 GB or more.

Actually it *is* using more than 1GB. Does that explain my problem? However (as 
I said) I do have at least a gig RAM to spare.

It seems as my subs list grows incrementally, this problem grows exponentially. 
FoxReplace is now useless for the large files it used to handle (albeit slowly) 
previously. From what you say, a fix will not only stop the freezes but improve 
performance overall. Sounds worth waiting for. It may even make a port to 
Android machines (with feeble CPU) feasible(?)

Original comment by sqgl2...@gmail.com on 11 Feb 2014 at 2:51

[GoogleCodeExporter]

I have found that Firefox is sluggish when it uses 1 GB or more RAM, even when 
I have 12 GB at home. And I mean sluggish for general use, not when using my 
addon. I suppose it has something to do with Firefox's memory management.

Original comment by marc.r...@gmail.com on 12 Feb 2014 at 9:26

[GoogleCodeExporter]

Hello, this happens to me with https://dl.dropbox.com/u/4967399/FoxReplace.json 
- its a page load script, but it happens on auto page reloads too. Examples: 
pretty much anything linked in https://news.google.pt/nwshp (ex: 
http://expresso.sapo.pt) and http://pplware.sapo.pt (ex: 
http://pplware.sapo.pt/informacao/engenheiros-informaticos-tem-desemprego-zero-e
m-portugal)

Sometimes it triggers a unresponsive script warning about foxreplace.js or some 
other webpage script, but not always, and it's unrelated to memory usage (ff at 
around 600MB will do), but it actually brings cpu usage down to next to zero, 
even freezing task manager, with a dualcore + 3gb mem, and win7 sp1 x64.

Original comment by C4rl0...@gmail.com on 6 Dec 2014 at 7:32

[GoogleCodeExporter]

Forgot to add, I'm using FR 0.16.1 and FF x64 Nightly 37.0.

Original comment by C4rl0...@gmail.com on 6 Dec 2014 at 7:34

[GoogleCodeExporter]

Thank you for the examples, they will be helpful.

Original comment by marc.r...@gmail.com on 6 Dec 2014 at 8:56