Log: Should write unicode characters into text file

[CyrusSUEN]

The dynamic log page can show unicode characters correctly including the Chinese ones but when viewing the full text log, non-English characters are corrupt.

Dynamic log:

Text log:

[dhowe]

Good point, this might be as converting the file to use a .html and including the correct charset tag:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

Please investigate

[dhowe]

@CyrusSUEN please test my fix c60bf77 before working on this (I am having trouble getting Chinese characters to show up in the log)

[CyrusSUEN]

test my fix c60bf77 before working on this

This fix doesn't work. Testing URL: https://www.google.com.hk/search?q=gym&ie=utf-8&oe=utf-8&gws_rd=cr&ei=KlmdVZTOCMWm0gSGlawQ

[CyrusSUEN]

I can confirm that the log text file is in UTF-8 encoding and can be correctly displayed by Atom editor or TextEdit of Mac.

But when opening the file via 'file://' protocol, Firefox has problem detecting the correct encoding: Bug 760050 - Character encoding auto-detect fails on UTF-8 text file

One way is like you mentioned to inject a HTML charset. (I'm trying this with PageMod) Another way is to include BOM in the text file. Here is a discussion whether or not to add BOM to UTF-8.

[CyrusSUEN]

inject a HTML charset with PageMod

https://github.com/dhowe/AdNauseam/compare/master...CyrusSUEN:iss384 This doesn't work and adding charset to the page after the document has been loaded is quite pointless as the browser has already decided how to render the text.

[dhowe]

This should be fixed. What I have done is allowed users with a previous (.txt) log to continue using it as before. New users and those who have deleted their logs get a new (.html) log with the proper encoding written at the top. To verify you must a) check the full log and make sure it works as before (without properly handling unicode characters), then b) delete your adnauseam-log.txt file (you can do this by disabling-logs in settings), then re-load the log and check that 1) it is an html file, and 2) unicode characters are rendered correctly.

[CyrusSUEN]

check that 1) it is an html file, and 2) unicode characters are rendered correctly.

Yes. Verified.

But disabling-logs in settings does not delete the log.txt for me. I have to manually delete the log.txt.

[dhowe]

Please recheck this after pulling my a654384

[CyrusSUEN]

recheck this after pulling

The no-deletion problem has been solved on Mac. But not on Windows. The following error happen when I'm viewing the log and toggled the disable log settings:

console.error: adnauseam:
  Message: [Exception... "Component returned failure code: 0x8052000e (NS_ERROR_
FILE_IS_LOCKED) [nsIFile.remove]"  nsresult: "0x8052000e (NS_ERROR_FILE_IS_LOCKE
D)"  location: "JS frame :: resource://gre/modules/commonjs/toolkit/loader.js ->
 resource://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js :: Logger<.
dispose :: line 59"  data: no]
  Stack:
    Logger<.dispose@resource://gre/modules/commonjs/toolkit/loader.js -> resourc
e://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js:59:7
UIManager<.registerEventHandlers/<@resource://gre/modules/commonjs/toolkit/loade
r.js -> resource://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/uiman.js:515:9

emit@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules
/commonjs/sdk/event/core.js:104:9
portEmit@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/mod
ules/commonjs/sdk/content/sandbox.js:343:7
emit@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules
/commonjs/sdk/event/core.js:104:9
onContentEvent/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource:/
/gre/modules/commonjs/sdk/content/sandbox.js:384:5
delay/<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modu
les/commonjs/sdk/lang/functional/concurrent.js:38:20
notify@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modul
es/commonjs/sdk/timers.js:40:9

[dhowe]

Any thoughts on why this is happening on Windows (permissions perhaps)? I've added some error-handling -- please recheck after pulling 93c1a03

[CyrusSUEN]

I first removed the log file and then pull the latest commit Here are the shell messages when I turn on and off the log settings: (The log file and the old log file data persist the whole time)

D:\Documents\GitHub\AdNauseam [master]> cfxp
Using binary at 'C:\Program Files (x86)\Mozilla Firefox\firefox.exe'.
Using profile at 'C:\Users\Cyrus\AppData\Roaming\Mozilla\Firefox\Profiles\ripltb
xm.cfxp'.
console.log: adnauseam: Created log(HTML): C:\Users\Cyrus\AppData\Roaming\Mozill
a\Firefox\Profiles\ripltbxm.cfxp\adnauseam-log.html
console.log: adnauseam: AdNauseam v1.302
console.log: adnauseam: [Options]
                        enabled=true
                        hideBadge=false
                        disableLogs=false
                        clearAdsWithHistory=false
                        disableOutgoingReferer=true
                        disableOutgoingCookies=true
                        disableIncomingCookies=true
                        platformVersion=39.0
                        production=true
                        version=1.302
console.log: adnauseam: Main::main() -> startup
console.log: adnauseam: AdParser: 0 ads (0 pending, 0 failed)
console.log: adnauseam: AdNauseam initialized
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: UI->show-log
WARNING: content window passed to PrivateBrowsingUtils.isWindowPrivate. Use isCo
ntentWindowPrivate instead (but only for frame scripts).
pbu_isWindowPrivate@resource://gre/modules/PrivateBrowsingUtils.jsm:25:14
nsBrowserAccess.prototype.openURI@chrome://browser/content/browser.js:15459:21
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: UI->disable-logs: true
console.log: adnauseam: Options.disableLogs=true
console.log: adnauseam: UI->close-log
console.warn: adnauseam: {}
console.warn: adnauseam: Logger.dispose() :: log still exists? true
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: Notify: No log available (AdNauseam is disabled or the '
disable-logs' preference is checked)
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: UI->disable-logs: false
console.warn: adnauseam: Logger: Reinitializing log
console.log: adnauseam: AdNauseam v1.302
console.log: adnauseam: [Options]
                        enabled=true
                        hideBadge=false
                        disableLogs=false
                        clearAdsWithHistory=false
                        disableOutgoingReferer=true
                        disableOutgoingCookies=true
                        disableIncomingCookies=true
                        platformVersion=39.0
                        production=true
                        version=1.302
console.log: adnauseam: Options.disableLogs=false

After toggling the disable-log settings a few times and then relaunch the browser:

D:\Documents\GitHub\AdNauseam [master]> cfxp
Using binary at 'C:\Program Files (x86)\Mozilla Firefox\firefox.exe'.
Using profile at 'C:\Users\Cyrus\AppData\Roaming\Mozilla\Firefox\Profiles\ripltb
xm.cfxp'.
console.error: adnauseam:
  Unable to create Logger: [xpconnect wrapped nsIFile]
  Message: [Exception... "Component returned failure code: 0x80520008 (NS_ERROR_
FILE_ALREADY_EXISTS) [nsIFile.create]"  nsresult: "0x80520008 (NS_ERROR_FILE_ALR
EADY_EXISTS)"  location: "JS frame :: resource://gre/modules/commonjs/toolkit/lo
ader.js -> resource://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js :
: Logger<.reset :: line 95"  data: no]
  Stack:
    Logger<.reset@resource://gre/modules/commonjs/toolkit/loader.js -> resource:
//adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js:95:9
Logger<.initialize@resource://gre/modules/commonjs/toolkit/loader.js -> resource
://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js:38:5
constructor@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/
modules/commonjs/sdk/core/heritage.js:146:23
@resource://gre/modules/commonjs/toolkit/loader.js -> resource://adnauseam-at-re
dnoise-dot-org/adnauseam/lib/ff/logger.js:365:18
evaluate@resource://gre/modules/commonjs/toolkit/loader.js:280:19
load@resource://gre/modules/commonjs/toolkit/loader.js:332:5
CuddlefishLoader/options<.load@resource://gre/modules/addons/XPIProvider.jsm ->
jar:file:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfx
p/extensions/adnauseam@rednoise.org.xpi!/bootstrap.js -> resource://gre/modules/
commonjs/sdk/loader/cuddlefish.js:79:18
require@resource://gre/modules/commonjs/toolkit/loader.js:625:16
@resource://gre/modules/commonjs/toolkit/loader.js -> resource://adnauseam-at-re
dnoise-dot-org/adnauseam/lib/main.js:3:16
evaluate@resource://gre/modules/commonjs/toolkit/loader.js:280:19
load@resource://gre/modules/commonjs/toolkit/loader.js:332:5
CuddlefishLoader/options<.load@resource://gre/modules/addons/XPIProvider.jsm ->
jar:file:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfx
p/extensions/adnauseam@rednoise.org.xpi!/bootstrap.js -> resource://gre/modules/
commonjs/sdk/loader/cuddlefish.js:79:18
main@resource://gre/modules/commonjs/toolkit/loader.js:740:10
run@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/
commonjs/sdk/addon/runner.js:145:19
startup/</<@resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/
modules/commonjs/sdk/addon/runner.js:86:7
Handler.prototype.process@resource://gre/modules/Promise.jsm -> resource://gre/m
odules/Promise-backend.js:867:23
this.PromiseWalker.walkerLoop@resource://gre/modules/Promise.jsm -> resource://g
re/modules/Promise-backend.js:746:7
this.PromiseWalker.scheduleWalkerLoop/<@resource://gre/modules/Promise.jsm -> re
source://gre/modules/Promise-backend.js:688:37
exports.Utils.yield@resource://gre/modules/addons/XPIProvider.jsm -> jar:file://
/C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/extension
s/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/bootstrap.js -> jar:file:///C:
/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/extensions/%
7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/lib/utils.js:382:12
INIParser.prototype.process@resource://gre/modules/addons/XPIProvider.jsm -> jar
:file:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/e
xtensions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/bootstrap.js -> jar:fi
le:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/exte
nsions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/lib/filterStorage.js:870:
7
exports.IO.readFromFile/onProgress@resource://gre/modules/addons/XPIProvider.jsm
 -> jar:file:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm
.cfxp/extensions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/bootstrap.js ->
 jar:file:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cf
xp/extensions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/lib/io.js:97:15
exports.IO.readFromFile/<@resource://gre/modules/addons/XPIProvider.jsm -> jar:f
ile:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/ext
ensions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/bootstrap.js -> jar:file
:///C:/Users/Cyrus/AppData/Roaming/Mozilla/Firefox/Profiles/ripltbxm.cfxp/extens
ions/%7Bd10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d%7D.xpi!/lib/io.js:182:11
TaskImpl_run@resource://gre/modules/Task.jsm:330:41
Handler.prototype.process@resource://gre/modules/Promise.jsm -> resource://gre/m
odules/Promise-backend.js:867:23
this.PromiseWalker.walkerLoop@resource://gre/modules/Promise.jsm -> resource://g
re/modules/Promise-backend.js:746:7
this.PromiseWalker.scheduleWalkerLoop/<@resource://gre/modules/Promise.jsm -> re
source://gre/modules/Promise-backend.js:688:37

console.warn: adnauseam: Logger: Reinitializing log
console.log: adnauseam: AdNauseam v1.302
console.log: adnauseam: [Options]
                        enabled=true
                        hideBadge=false
                        disableLogs=false
                        clearAdsWithHistory=false
                        disableOutgoingReferer=true
                        disableOutgoingCookies=true
                        disableIncomingCookies=true
                        platformVersion=39.0
                        production=true
                        version=1.302
console.log: adnauseam: Main::main() -> startup
console.log: adnauseam: AdParser: 0 ads (0 pending, 0 failed)
console.log: adnauseam: AdNauseam initialized
console.log: adnauseam: No-ads on page, showing 0 recent
console.log: adnauseam: UI->show-log

[dhowe]

This is because the original delete failed (on Windows), then we try to recreate the logfile, which already still exists... Can you research why we cannot delete a file from the addon on windows?

[dhowe]

@cyrus please run the full set of tests (the log will NOT be deleted anymore) on both Windows+Mac

[CyrusSUEN]

run the full set of tests (the log will NOT be deleted anymore) on both Windows

Before test: removed the log file and @ commit e40de39857c9eab2cb39f918f615c2eefec00e0f Test method: Toggled the disable-log settings a few times and then relaunch the browser (same result):

console.error: adnauseam:
  Unable to create Logger: [xpconnect wrapped nsIFile]
  Message: [Exception... "Component returned failure code: 0x80520008 (NS_ERROR_
FILE_ALREADY_EXISTS) [nsIFile.create]"  nsresult: "0x80520008 (NS_ERROR_FILE_ALR
EADY_EXISTS)"  location: "JS frame :: resource://gre/modules/commonjs/toolkit/lo
ader.js -> resource://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js :
: Logger<.reset :: line 95"  data: no]

[dhowe]

You may need to delete the log in your windows profile before testing. Either way, please retest with e40de39

[CyrusSUEN]

need to delete the log in your windows profile before testing

Yes. I did that before the test. And conducted the test multiple times with the same result.

[dhowe]

with the new commit?

[CyrusSUEN]

with the new commit?

Yes. The same. I will research how to fix this on Windows.

[dhowe]

same on mac, or just windows?

[CyrusSUEN]

same on mac, or just windows?

This happens on both Mac and Windows.

Before test: removed the log file Test method: Open the log and toggled the disable-log settings a few times then relaunched the browser:

console.error: adnauseam:
  Unable to create Logger: [xpconnect wrapped nsIFile]
  Message: [Exception... "Component returned failure code: 0x80520008 (NS_ERROR_
FILE_ALREADY_EXISTS) [nsIFile.create]"  nsresult: "0x80520008 (NS_ERROR_FILE_ALR
EADY_EXISTS)"  location: "JS frame :: resource://gre/modules/commonjs/toolkit/lo
ader.js -> resource://adnauseam-at-rednoise-dot-org/adnauseam/lib/ff/logger.js :
: Logger<.reset :: line 95"  data: no]

Let's call this e2 (error 2). e1 was the error that on Windows it could not delete the log file.

[dhowe]

ok, I will look at it...

[dhowe]

I can't reproduce this on OS X. @mushon? @CyrusSUEN I wonder if your system is in some strange state due to the manual deletes...

[mushon]

I followed @CyrusSUEN's instructions but I couldn't recreate it on Mac either

[CyrusSUEN]

To reproduce:

1. locate and delete the 'adnauseam-log' file (whether it's .txt or .html) in the profile folder
2. open Firefox and the dynamic log page
3. toggle the 'disable-log' settings in ADN menu on (should be off by default) and then off
4. restart Firefox

E2 should appear with the above console messages

[mushon]

that's what I did, but E2 didn't appear in my console

[dhowe]

same here

[CyrusSUEN]

No matter on Mac or Windows and to use a new profile or not. I can reproduce E2 following the above steps.

And to 'solve' E2 on my machines I can simply delete the log file from the profile folder and do not toggle the disable-log setting.

I think this might be related to the file permission. So I ran $ ls -l on the profile folder and here is the status of the log file on my Mac

-rw-r--r--@  1 cyrussuen  staff      1452 Jul 10 18:21 adnauseam-log.html

Can I have your status on this file?

[dhowe]

what about the parent directory?

[CyrusSUEN]

what about the parent directory?

drwx------@ 48 cyrussuen  staff  1632 Jul 10 18:21 05rhodfg.cfxo

[dhowe]

directory: drwxr-xr-x file: -rw-r--r--

[CyrusSUEN]

directory: drwxr-xr-x

I did a chmod 755 on the profile folder and the problem is still there.

[dhowe]

try chmod -R 755 on the entire profile

[CyrusSUEN]

try chmod -R 755 on the entire profile

Same error

[CyrusSUEN]

Actually this ticket has been solved and we should move the discussion to https://github.com/dhowe/AdNauseam/issues/389