AutoExport doest not work for archive.org

[yuisong]

Hi, When I want to generate the har file of https://archive.org/, which is a library of history of millions of website pages, there are always some errors. I use firebug-3.0.0-alpha.12.xpi+FF for mac 37.0.2.

1. When I generated the har files manually, the files cannot be analyzed by the HAR archive viewer, no matter the home page ( https://archive.org/ ) or the specific history pages I want to query. eg: https://web.archive.org/web/20150425114317/https://www.yahoo.com/
2. When I use Selenium + FF + firebug-3.0.0-alpha.12.xpi to generate har files automatically, the UI looks different and the output of the home page, archive.org, cannot be analyzed by the viewer, either. More worse, no har file was generated for any history pages. eg: https://web.archive.org/web/20070802094218/http://www.google.com/

Here is the screenshot of firebug without Selenium (I have try to turn it into an English version, but I don't know how and finally can only provide a Chinese version. I hope you can understand what I mean)

Screenshot of firebug when using Selenium. As you can see, no export panel here. However, I can use my script to generate har files for other website, eg: www.google.com. Just no output for history pages from archive.org :(

The output of the viewer when analysing the home page:

And here is some of my code:

    // Set default Firefox preferences

    profile.setPreference("app.update.enabled", false);

   domain = "extensions.firebug@software.joehewitt.com.";            

   // Set default Firebug preferences       

    profile.setPreference(domain + "allPagesActivation", "on");

    profile.setPreference(domain + "netexport.alwaysEnableAutoExport", true);

    profile.setPreference(domain + "netexport.showPreview", false);

    profile.setPreference(domain + "netexport.autoExportToFile", true);

    profile.setPreference(domain + "netexport.saveFiles", true);

    profile.setPreference(domain + "netexport.defaultLogDir", "~/project/har");

    profile.setPreference("devtools.toolbox.selectedTool", "netmonitor"); //The net panel is selected by default

    profile.setPreference("devtools.netmonitor.enabled", true);  //Net panel is enabled by default

    //set the file name

    Date date = calendar.getTime(); //the date of history page you want to get

    DateFormat format = new SimpleDateFormat("yyyyMMdd");

    String dateStr=format.format(date);

    String fileName=websiteName+"+"+dateStr;

    profile.setPreference(domain + "netexport.defaultFileName", fileName);

3.What will happen if the url I getting is redirected to another url? Will the har file of another pages be generated? Or the redirection will be included in the har log file?

Can you have a look at it?

Many thx!

Yui

[yuisong]

And today when I try to use firebug-2.0.8-fx.xpi+netExport-0.9b7.xpi+selenium to generate har files for history pages, they work smoothly, except that I cannot set names for the files. The all named web.archive.org+yyyy-mm-dd.har, hence I don't know which history page is which.

[JosephCastro]

• You can use the JS API ? (window.NetExport.triggerExport(secretToken) )
• You trying to generate HAR files using a proxy ?

Edit

You need to set other settings for Firebug3 -alpha - x , see about:config and search : netexport .

[yuisong]

You can use the JS API ? (window.NetExport.triggerExport(secretToken) ) ------>I am using java to write the script and I have tried to use the JS API, but it did not work.

You trying to generate HAR files using a proxy ? ------>And yes, because archive.org is blocked in China, so I have to use a VPN to get access to it.

You need to set other settings for Firebug3 -alpha - x , see about:config and search : netexport . ------>As the code I have mentioned before, I have set some default value. What else should I set?

Many thanks! Yui

[fflorent]

ping @janodvarko ?

Florent

[JosephCastro]

@yuisong set this:

• autoExportToServer : False
• defaultLogDir : '/path/to/a/folder/to/save/har'
• secretToken : 'string-token' (with this you can access to the JS API to trigger the HAR export)

And maybe timeout if you don't want to wait so many time (you have 60 seconds in miliseconds)

NOTE : I have troubles with FF40-Nightly and Firebug3-Alpha-12 and Proxy, I can't get the HAR file properly, see this for more information.

[janodvarko]

Can anyone test my changes? Do you want me to generate XPI?

Or you can: git clone https://github.com/firebug/firebug.next.git cd firebug.next jpm xpi

Honza

[yuisong]

Or yes, please generate the XPI. I have not installed npm and jpm, so if possible, please generate it for me. I will test it.

Many thanks! Yui

[janodvarko]

Here is the XPI http://www.janodvarko.cz/temp/firebug/firebug-3.0.0-alpha.12.389.xpi

Honza

[janodvarko]

HAR export feature is now built in Firefox and there is also HARExportTrigger that exposes HAR API to the content. Please report any related issues in http://bugzilla.mozilla.org/ or https://github.com/firebug/har-export-trigger/

Honza