search

jimmejardine / qiqqa-open-source

The open-sourced version of the award-winning Qiqqa research management tool for Windows
GNU General Public License v3.0
378 stars 65 forks source link

Unable to flush xulrunner cache - sniffer #330

Open aterhorst opened 3 years ago

aterhorst commented 3 years ago

I, like many others, am encountering reCAPTCHA and other weird issues with Google Scholar. The built-in xulrunner browser does not support reCAPTCHA requests. I also cannot flush the browser cache. Can you please update the built-in browser and add controls to flush the browser cache and reset cookies? Xulrunner stopped being developed in 2015.

aterhorst commented 3 years ago

image

This the latest weird issue. I got around this by flushing the browser cache on my main web browser.

GerHobbelt commented 3 years ago

overlap with #289, #225, #113, #2

😰 I know. And do realize the urgency of this. (The xulrunner core is comparable to Firefox version 33 .. 36 there-about. Antique in web/internet terms. Transition is unfortunately not an easy task, given what the Qiqqa Sniffer does and needs to do its job. 😰

As always, thank you for reporting and detail.

Section below is just FYI so you and others can see what's been going on and where we are right now. While it may be technical, I hope it sheds some light on the proceedings.

Don't hesitate to ask questions or give critique.

Current status is: near end of viability research. I've looked into CEF+C# based solutions (CEFSharp, CEFGlue, ...) where I dearly wanted to pick something that got me at least a toe-hold into cross-platform support (i.e. Linux/UNIX support: #215) so I wouldn't have to undertake this particular endeavour twice. (once to modernize the windows-only version, then once more to yet another library to make it work on both Unixes (Linuxes mostly -- BSD, anyone?) and Windows (and maybe Macs, but I don't have Apple around and won't foot that bill. Anyway.)

CEFSharp/Glue/etc. look great at the surface, until you get down to the nitty-gritty of the needs of Qiqqa/Sniffer. (ability to hook into and redirect/tee/split http request responses so we can get our grubby paws on the "downloaded data" before the xulrunner/browser does + full access to the rendered DOM so we can sniff/"scrape" the page for links and content: the BibTeX auto-fetch behaviour.

CEFSharp is Windows only as it's got C++/CLI, which is beautiful stuff but horribly non-portable as nobody offers this for Linuxes, which leaves the P/Invoke based approaches. Not the nicest way to talk across language boundaries in software, IMO. Alas.

Xilium was right out after the initial tests I did last year as it wasn't doing anything but crapping on my dev system when I applied a bit of stress to it. May have been me, but it's got to work here if I want to have a chance at pulling it into Qiqqa and keeping it all alive.

CEFGlue was a bother too, but they did something I haven't identified exactly, that made it far less brittle on my machine since Q4 2020. CEFGlue purportedly supports Linux. (Which leaves the elephant in the room: Qiqqa's WPF-based UI: WPF and Linux don't mix. (And Avalonia is not WPF)

CEFGlue also has an answer for the 32-/64-bit .NET challenge, so that would leave #289 pending until I finally kick out another veritable antique: that's our Full Text Search engine Lucene.NET at #23.

Given the amount of hassle I had with the initial trials with these CEF+C# fella's, plus the whole WPF @#$%^censored@#$% "fun", I've been considering to directly use CEF itself instead of a (critical, yet lower git(hub) activity) C# wrapper for it. That implies doing either some UI work in C++ (not the brightest idea ever for cross-platform 'koff' 'koff') or consider taking a ride on a solution like electron or similar goods (but more geared towards talking to C# code, that could/would produce data for the UI, e.g. Chromely). That means re-doing the UI in HTML/JS/CSS, which would 'solve' #215 almost by default, but can reasonably only be a long term goal given the effort required for that transition. (I want to move away from WPF anyway, but time, etc....: which makes this a long term goal.)

Meanwhile Microsoft put in some noticeable effort into WebView2 + Chrome~=Edge last year and I have to consider your (and really everyone's) problem with the current Qiqqa Sniffer, so I must do what I initially didn't want/liked to do: the two stage process: first fix this for Windows only. Which is why WebView2 is a viable option and the last one considered. Then do it again later, now with Linux in mind.

So the current thinking here is this:

Incidentally, if anyone likes technical challenges, I've got some on offer to sink your teeth in. 😉 (This stuff needs to work in "production environments" (i.e. actual users) so a PoC demo/example code isn't the finish line. 😜 )

aterhorst commented 3 years ago

Thank you for all the hard work. I take my hat off to the OSS community. Qiqqa is a complex piece of software. One has to wonder at what stage should one consider replacing it with a completely new platform? That thought has crossed your mind many a time, I am sure.

From: Ger Hobbelt @.> Reply to: jimmejardine/qiqqa-open-source @.> Date: Thursday, 27 May 2021 at 9:49 am To: jimmejardine/qiqqa-open-source @.> Cc: "Terhorst, Andrew (Data61, Sandy Bay)" @.>, Author @.***> Subject: Re: [jimmejardine/qiqqa-open-source] Unable to flush xulrunner cache - sniffer (#330)

overlap with #289https://github.com/jimmejardine/qiqqa-open-source/issues/289, #225https://github.com/jimmejardine/qiqqa-open-source/issues/225, #113https://github.com/jimmejardine/qiqqa-open-source/issues/113, #2https://github.com/jimmejardine/qiqqa-open-source/issues/2

😰 I know. And do realize the urgency of this. (The xulrunner core is comparable to Firefox version 33 .. 36 there-about. Antique in web/internet terms. Transition is unfortunately not an easy task, given what the Qiqqa Sniffer does and needs to do its job. 😰

As always, thank you for reporting and detail.

Section below is just FYI so you and others can see what's been going on and where we are right now. While it may be technical, I hope it sheds some light on the proceedings.

Don't hesitate to ask questions or give critique.

Current status is: near end of viability research. I've looked into CEF+C# based solutions (CEFSharp, CEFGlue, ...) where I dearly wanted to pick something that got me at least a toe-hold into cross-platform support (i.e. Linux/UNIX support: #215https://github.com/jimmejardine/qiqqa-open-source/issues/215) so I wouldn't have to undertake this particular endeavour twice. (once to modernize the windows-only version, then once more to yet another library to make it work on both Unixes (Linuxes mostly -- BSD, anyone?) and Windows (and maybe Macs, but I don't have Apple around and won't foot that bill. Anyway.)

CEFSharp/Glue/etc. look great at the surface, until you get down to the nitty-gritty of the needs of Qiqqa/Sniffer. (ability to hook into and redirect/tee/split http request responses so we can get our grubby paws on the "downloaded data" before the xulrunner/browser does + full access to the rendered DOM so we can sniff/"scrape" the page for links and content: the BibTeX auto-fetch behaviour.

CEFSharp is Windows only as it's got C++/CLI, which is beautiful stuff but horribly non-portable as nobody offers this for Linuxes, which leaves the P/Invoke based approaches. Not the nicest way to talk across language boundaries in software, IMO. Alas.

Xilium was right out after the initial tests I did last year as it wasn't doing anything but crapping on my dev system when I applied a bit of stress to it. May have been me, but it's got to work here if I want to have a chance at pulling it into Qiqqa and keeping it all alive.

CEFGlue was a bother too, but they did something I haven't identified exactly, that made it far less brittle on my machine since Q4 2020. CEFGlue purportedly supports Linux. (Which leaves the elephant in the room: Qiqqa's WPF-based UI: WPF and Linux don't mix. (And Avalonia is not WPF)

CEFGlue also has an answer for the 32-/64-bit .NET challenge, so that would leave #289https://github.com/jimmejardine/qiqqa-open-source/issues/289 pending until I finally kick out another veritable antique: that's our Full Text Search engine Lucene.NET at #23https://github.com/jimmejardine/qiqqa-open-source/issues/23.

Given the amount of hassle I had with the initial trials with these CEF+C# fella's, plus the whole WPF @#$%^censored@#$% "fun", I've been considering to directly use CEF itself instead of a (critical, yet lower git(hub) activity) C# wrapper for it. That implies doing either some UI work in C++ (not the brightest idea ever for cross-platform 'koff' 'koff') or consider taking a ride on a solution like electron or similar goods (but more geared towards talking to C# code, that could/would produce data for the UI, e.g. Chromely). That means re-doing the UI in HTML/JS/CSS, which would 'solve' #215https://github.com/jimmejardine/qiqqa-open-source/issues/215 almost by default, but can reasonably only be a long term goal given the effort required for that transition. (I want to move away from WPF anyway, but time, etc....: which makes this a long term goal.)

Meanwhile Microsoft put in some noticeable effort into WebView2 + Chrome~=Edge last year and I have to consider your (and really everyone's) problem with the current Qiqqa Sniffer, so I must do what I initially didn't want/liked to do: the two stage process: first fix this for Windows only. Which is why WebView2 is a viable option and the last one considered. Then do it again later, now with Linux in mind.

So the current thinking here is this:

Incidentally, if anyone likes technical challenges, I've got some on offer to sink your teeth in. 😉 (This stuff needs to work in "production environments" (i.e. actual users) so a PoC demo/example code isn't the finish line. 😜 )

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/jimmejardine/qiqqa-open-source/issues/330#issuecomment-849177809, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABYGZCYIXYVQR4DAX5LMB3DTPV5PXANCNFSM45ONGX4Q.