Closed AnQueth closed 3 years ago
champnic
commented
3 years ago Hey @AnQueth - can you give some more context about when you are running this? Are you running this many times in succession, or does it block when you call it from a specific event or stack?
AnQueth
commented
3 years ago so it's a scraping situation. i automate it with the ExecuteScriptAsync it will run many times in succession. i want to say around 8 calls and then it can freeze. the JavaScript executes, but the await never returns. when it breaks, I will have tasks left over in the task window in vs. few of the scripts look like this:
string selectOption = " function modelChange(){ try{ var el = document.getElementById(\"" + el + "\");" +
"var op = el.options[" + CModel + "]; " +
"$('#" + el + "').val(op.value);" +
"$('#" + el + "').trigger('change');" +
"return 'good';" +
"} catch(e){ return 'bad'; }}\r\n" +
"modelChange();";
var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption);
ukandrewc
commented
3 years ago @AnQueth I don't want to suggest that there isn't an issue, just wanted to confirm that it doesn't always fail with multiple calls.
I've just successfully called ExecuteScriptAsync to modify the DOM, 1032 times in 1.7s, without any failures.
AnQueth
commented
3 years ago yes, i don't know why it will not let the await finish but the js runs.
ukandrewc
commented
3 years ago Where are you calling this from, are you inadvertently blocking it?
I've just tried some adhoc tests, to call 100 times async, then 100 times sync, then 100 times async again, with no issues, using this page: https://www.w3.org/WAI/UA/2002/06/thead-test
Script to style border of table:
for (const table of document.getElementsByTagName("table")) {
table.style.tableLayout = "fixed"
table.style.border = "2px solid " + getColour()
for (const row of table.rows) {
for (const cel of row.cells) {
cel.style.border = "2px solid " + getColour()
}
}
}
function getColour() {
return choose(1 + Math.random() * 5, ["red", "green", "gold", "blue", "orange"])
}
function choose(n, l) {
return l[Math.round(n) - 1]
}Then on the .Net side
Sub Btn_Click()
TestAsync()
TestSync()
TestAsync()
End Sub
Private Async Sub TestSync()
For i = 1 To 100
Dim r = Await Web.CoreWebView2.ExecuteScriptResourceAsync(JS)
Next
End Sub
Private Sub TestAsync()
For i = 1 To 100
Web.CoreWebView2.ExecuteScriptResourceAsync(JS)
Next
End Sub@AnQueth I wonder if the javascript itself is hanging or blocking? Is there anything in your javascript that could possibly block? When you run into this issue, if you open the devtools on that page, can you manually run your script in the console successfully?
Also, thanks @ukandrewc for the extra testing here and confirming it isn't a wider problem (potentially)!
ukandrewc
commented
3 years ago @champnic No problem. We use ExecutescriptAsync a lot, so it's in our interest to see if it can be broken ;-)
AnQueth
commented
3 years ago here's the whole code. it's a messy mess because it's just hacky thing. the jist of it is login by using the form and submitting so the tokens google work. when the webpacked file is downloaded, intercept it and expose jquery globably. then manipulate the combo boxes which use jqueryui select2 plugin. when ajax request completes, run my selection code for the drop downs. then submit the form and start paging the results if it has pages then repeat everything but login when out of pages.
i did have this before, and it would block to. there is nothing there should block.
string selectOption = "document.documentElement.outerHTML"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); var html = Regex.Unescape(r);
`
using Azure.Messaging.ServiceBus;
using Microsoft.Web.WebView2.Core;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
namespace autoastat
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
ServiceBusClient serviceBusClient = new ServiceBusClient("E");
private const string Dir = "f:\\scrape";
private int _currentMake = 1;
private int _currentModel = 1;
private string _currentUri = "";
private bool _loggedin;
private Uri _referer;
private string _username = "login";
private string _password = "7Fpassword";
private ServiceBusSender _sbc;
private int CMake
{
get => _currentMake;
set
{
_currentMake = value;
Save();
}
}
private int CModel
{
get => _currentModel;
set
{
_currentModel = value;
Save();
}
}
private void Save()
{
var ob = new
{
CModel = _currentModel,
CMake = _currentMake,
Page = _currentUri
};
var d = JsonConvert.SerializeObject(ob);
File.WriteAllText("data.json", d);
}
private void Load()
{
if (File.Exists("data.json"))
{
var d = JsonConvert.DeserializeObject<dynamic>(File.ReadAllText("data.json"));
_currentMake = (int)d["CMake"].Value;
_currentModel = (int)d["CModel"].Value;
_currentUri = d["Page"].Value;
}
}
public MainWindow()
{
AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException;
InitializeComponent();
}
private void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
{
}
private async void Grid_Loaded(object sender, RoutedEventArgs e)
{
_sbc = serviceBusClient.CreateSender("scrape");
Load();
var env = await CoreWebView2Environment.CreateAsync();
await web.EnsureCoreWebView2Async(env);
web.CoreWebView2.ProcessFailed += CoreWebView2_ProcessFailed;
web.CoreWebView2.OpenDevToolsWindow();
//web.CoreWebView2.AddWebResourceRequestedFilter("https://autoastat.com/build/vehicle/main.1e19bf9f.js",
// Microsoft.Web.WebView2.Core.CoreWebView2WebResourceContext.All);
web.CoreWebView2.AddWebResourceRequestedFilter("*", Microsoft.Web.WebView2.Core.CoreWebView2WebResourceContext.All);
web.CoreWebView2.WebResourceRequested += this.CoreWebView2_WebResourceRequested;
web.CoreWebView2.DOMContentLoaded += this.CoreWebView2_DOMContentLoaded;
web.CoreWebView2.NavigationStarting += this.CoreWebView2_NavigationStarting;
web.CoreWebView2.WebResourceResponseReceived += this.CoreWebView2_WebResourceResponseReceived;
web.CoreWebView2.NavigationCompleted += this.CoreWebView2_NavigationCompleted;
web.CoreWebView2.Navigate("https://autoastat.com/en/login");
}
private void CoreWebView2_ProcessFailed(object sender, Microsoft.Web.WebView2.Core.CoreWebView2ProcessFailedEventArgs e)
{
}
private void CoreWebView2_NavigationStarting(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationStartingEventArgs e)
{
}
private async void CoreWebView2_WebResourceRequested(object sender, Microsoft.Web.WebView2.Core.CoreWebView2WebResourceRequestedEventArgs e)
{
if (_referer != null)
e.Request.Headers.SetHeader("Referer", _referer.ToString());
if (e.Request.Uri.Contains("main") && e.Request.Uri.Contains("js"))
{
HttpClient cl = new HttpClient();
var d = e.GetDeferral();
var data = await cl.GetStringAsync(e.Request.Uri);
const string s = "(function(e){n(\"8h6d\");";
data = data.Insert(data.IndexOf(s) + s.Length, "window.$=e;");
Stream st = new MemoryStream(Encoding.UTF8.GetBytes(data));
e.Response = web.CoreWebView2.Environment.CreateWebResourceResponse(st, 200, "OK", $"Content-Type: application/javascript");
d.Complete();
}
}
private async void CoreWebView2_DOMContentLoaded(object sender, Microsoft.Web.WebView2.Core.CoreWebView2DOMContentLoadedEventArgs e)
{
if (web.Source.PathAndQuery.EndsWith("en/"))
{
}
}
private async void CoreWebView2_WebResourceResponseReceived(object sender, Microsoft.Web.WebView2.Core.CoreWebView2WebResourceResponseReceivedEventArgs e)
{
if (e.Request.Uri.Contains("en/form?"))
{
try
{
var s = await e.Response.GetContentAsync();
var buf = new byte[s.Length];
s.Read(buf, 0, buf.Length);
var html = Encoding.UTF8.GetString(buf);
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
if (!Directory.Exists(Dir))
Directory.CreateDirectory(Dir);
foreach (var node in doc.DocumentNode.SelectNodes("//div[@class='block']"))
{
//await File.WriteAllTextAsync(
// $"{Dir}\\{CMake}_{CModel}_{DateTimeOffset.UtcNow.Ticks}", node.OuterHtml);
var a = node.SelectSingleNode(".//a").GetAttributeValue("href", "");
var al = "https://autoastat.com" + a;
var fn = GetFileName(al);
await _sbc.SendMessageAsync(new ServiceBusMessage(al));
using (var fs = File.Open(Dir + "\\" + fn, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
using (var sr = new StreamWriter(fs))
{
sr.WriteLine(al);
}
}
//File.AppendAllText($"{Dir}\\{fn}", al + "\r\n");
}
HtmlAgilityPack.HtmlNode? paging = null;
var pn = doc.DocumentNode.SelectNodes("//a[@class='page-link']");
if (pn != null)
{
paging = pn.LastOrDefault();
}
if (paging == null)
{
await Task.Delay(new Random().Next(60000, 120000));
_currentUri = "";
Save();
CModel++;
_referer = null;
web.Source = new Uri("https://autoastat.com/en/");
}
else
{
_referer = web.Source;
string u = "https://autoastat.com" +
paging.GetAttributeValue("href", "").Replace("&", "&");
var src = HttpUtility.ParseQueryString(_referer.Query).Get("page");
var ns = HttpUtility.ParseQueryString(new Uri(u).Query).Get("page");
if (!int.TryParse(src, out var old))
{
old = 0;
}
if (!int.TryParse(ns, out var news))
{
}
if (news > old)
{
await Task.Delay(new Random().Next(60000, 120000));
web.Source = new Uri(u);
_currentUri = u;
Save();
}
else
{
await Task.Delay(new Random().Next(60000, 120000));
CModel++;
_referer = null;
_currentUri = "";
Save();
web.Source = new Uri("https://autoastat.com/en/");
}
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
web.Reload();
}
}
if (e.Request.Uri.Contains("main") && e.Request.Uri.EndsWith(".js"))
{
//var s = await e.Response.GetContentAsync();
//var buf = new byte[s.Length];
//s.Read(buf, 0, buf.Length);
//string g = Encoding.UTF8.GetString(buf);
}
if (e.Request.Uri.Contains("/en/make"))
{
try
{
var s = await e.Response.GetContentAsync();
// _makes = await JsonSerializer.DeserializeAsync<Make>(s);
var buf = new byte[s.Length];
s.Read(buf, 0, buf.Length);
var _makes = JsonConvert.DeserializeObject<Make>(Encoding.UTF8.GetString(buf));
Task.Delay(1000).ContinueWith(async (t) =>
{
Application.Current.Dispatcher.Invoke(async () =>
{
string el = "select-make";
string selectOption = " try{ var el = document.getElementById(\"" + el + "\");" +
"var op = el.options[" + CMake + "]; " +
"$('#" + el + "').val(op.value);" +
"$('#" + el + "').trigger('change');" +
"} catch(e){alert(e);}";
var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption);
});
});
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
}
}
if (e.Request.Uri.Contains("en/model"))
{
Task.Delay(500).ContinueWith(async (t) =>
{
Application.Current.Dispatcher.Invoke(async () =>
{
string el = "select-model";
string selectOption = " function modelChange(){ try{ var el = document.getElementById(\"" + el + "\");" +
"var op = el.options[" + CModel + "]; " +
"$('#" + el + "').val(op.value);" +
"$('#" + el + "').trigger('change');" +
"return 'good';" +
"} catch(e){ return 'bad'; }}\r\n" +
"modelChange();";
var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption);
if (r == "\"bad\"")
{
CMake++;
CModel = 1;
web.Reload();
//web.Source = new Uri("https://autoastat.com/en/");
}
});
});
}
if (e.Request.Uri.Contains("en/date"))
{
var s = await e.Response.GetContentAsync();
// _makes = await JsonSerializer.DeserializeAsync<Make>(s);
var buf = new byte[s.Length];
s.Read(buf, 0, buf.Length);
string ds = "";
try
{
ds = Encoding.UTF8.GetString(buf);
var dates = JsonConvert.DeserializeObject<Dates>(ds);
var dts = dates.results.Select(z => DateTime.Parse(z)).Where(z =>
z >= DateTime.Today.AddDays(-14)).OrderByDescending(p => p).ToArray();
if (dts.Count() == 0)
{
dts = dates.results.Select(z => DateTime.Parse(z)).OrderByDescending(p => p).ToArray();
}
var old = dts.LastOrDefault();
var n = dts.FirstOrDefault();
if (old == default)
old = n;
if (old == default && n == default)
{
return;
}
Task.Delay(20000).ContinueWith(async (t) =>
{
Application.Current.Dispatcher.Invoke(async () =>
{
string selectOption = " try{ $('#select-date-from').val('" + old.ToString("yyyy-MM-dd") + "') ; " +
"$('#select-date-to').val('" + n.ToString("yyyy-MM-dd") + "') ; " +
"$('#search').click(); " +
"} catch(e){console.log(e);}";
var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption);
});
});
}
catch
{
Debug.WriteLine(ds);
CModel++;
web.Reload();
}
}
}
private async void CoreWebView2_NavigationCompleted(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationCompletedEventArgs e)
{
if (_loggedin)
{
_loggedin = false;
if (_currentUri != string.Empty)
{
web.Source = new Uri(_currentUri);
return;
}
else
{
_currentUri = web.Source.ToString();
Save();
}
}
if (web.Source.PathAndQuery.EndsWith("login"))
{
string login = "document.getElementById('_username').value = '" + _username + "';" +
"document.getElementById('_password').value = '" + _password + "';" +
" document.querySelectorAll(\"button[type='submit']\")[2].click();";
await web.CoreWebView2.ExecuteScriptAsync(login);
_loggedin = true;
}
}
private string GetFileName(string al)
{
int c = 0;
foreach (byte x in al)
{
c += x;
}
int r = c % 1;
return r.ToString() + ".urls";
}
}
}@AnQueth Unfortunately, you've got too much going on there, to be able to offer anything constructive.
hmurty
commented
3 years ago Hi, I am encountering a similar issue. I just checked my code with your comment in mind re: deadlock with Navigation completed. These events happen very early on as the page loads. The issue happens much later during the 50th refresh.
I reduced my code to multiple refresh clicks (using ExecuteScriptAsync) on a web page. Doing this out of the system timer tick. Calling an async function that has the await EexecuteScriptAsync call. The tick happens every 2 seconds. I also tried it for up to 6 seconds with similar results.
This async function sets a flag indicating await has completed. This way the next function call starts after the previous await has completed.
The refreshes stop happening after about 50 of them. For 2 second intervals at tick number 50, for 6s at tick number 48. I can manually refresh the page after this freeze of ExecuteScriptAsync. Tried over 50 manual refreshes 2 seconds apart and it continues to work.
The interesting thing is this piece of code worked for over 6 hours last Friday, May 28th. But stopped working on Tuesday June 1st. I was using VS 2019 16.10. Reverted back to 16.9.6 and saw the same issue. Tried on two machines with similar results.
Do not know if the webView2 uses whatever the latest Edge browser is on this Windows 10 machine. If the browser updates without my knowledge that could be one reason.
Here is the function public async void runJSScriptAsyncGSQRefreshAsync(int anchorCount) { activeJSAwaitFunctionCount++; backFromJSScriptAsyncGSQRefresh = false; await webViewMasterWB.ExecuteScriptAsync($"var anchors = document.getElementsByTagName('a'); anchors[{anchorCount}].click();"); backFromJSScriptAsyncGSQRefresh = true; activeJSAwaitFunctionCount--; }
Here is the caller:
if (backFromJSScriptAsyncGSQRefresh == true)
{
Debug.WriteLine("line 3945 launching GSQ refresh");
runJSScriptAsyncGSQRefreshAsync(anchorCount);
}
This is what seems to be the sequence of events; 1) From timer tick the async function is called 2) The function executes the ExecuteScriptAsync inside it 3) This executes a piece of JavaScript code that clicks a refresh button on the page 4) The await then returns: it clicked the refresh button, task completed. 5) The embedded JavaScript code tied to this button click fetches updated data from the server. It is a single page situation. The page does not reload: just one piece of data on the page is updated, 6) This may hang after a while with no response from server but this is not why my await ExecScrAsync does not return. 7) I stop seeing Debug.WriteLine("line 3945 launching GSQ refresh"); in output 8) meaning (backFromJSScriptAsyncGSQRefresh is false which means await has not returned
This makes me think that this is an issue with webView2. Any insight would be much appreciated. This could be easily repeated with any SPA web page that has a refresh button on it.
Thanks a bunch!
AnQueth
commented
3 years ago yes, it ran fine 2 weeks ago for me to.
ukandrewc
commented
3 years ago @hmurty Do you get the same if your JS calls location.reload()?
hmurty
commented
3 years ago ukandrewc thank you for suggestion, will try and get back to you in a day or two.
hmurty
commented
3 years ago @ukandrewc location.reload() takes a longer time. It would be hard to get new data every two seconds. There are other parts of the code where loading a new page takes over 10 seconds. Why I prefer to use minor changes on a page for quick updates. As an experiment I can try to reload when stuck. If this worked what would it indicate? It would not be a preferred solution. It would be very nice to get back to the situation on May 28th when the current code just ran without a hiccup. Thanks!
ukandrewc
commented
3 years ago @hmurty Sorry, I was only asking because I've tried a timer test, like yours, but using location.reload(), not an AJAX call.
hmurty
commented
3 years ago @ukandrewc location.reload() does not activate embedded JavaScript code but having .NET code click on a refresh button initiates the embedded handler for the click. The webView2 browser object has to manage JavaScript from ExecuteScriptAsync and also the embedded JavaScript. There may be an issue there if the external JS comes on top of the embedded JS that has not yet finished. This is why I increased the time between two timer ticks but results did not change.
ukandrewc
commented
3 years ago @hmurty are you able to share the url you are using?
hmurty
commented
3 years ago @ukandrewc it is an account I log into. Will try to find something similar that is public in a day or two. It is stock quotation data. Probably Yahoo Finance has something similar. Even if it is delayed data it will be changing regularly. If there is a hang up after so many refreshes this would point to .NET. Thank you for your interest!
champnic
commented
3 years ago @hmurty @anqueth - What version of the runtime are you seeing this on? We recently had an update to the WebView2 Runtime on May 25th-ish, and I wonder if this caused something to regress.
AnQueth
commented
3 years ago Everything worked great before memorial day weekend. After is when I noticed the problem so the timing does match the upgrade.
hmurty
commented
3 years ago 
champnic
commented
3 years ago Thanks for the info both! @hmurty That's the version of the SDK package. For the runtime, you can go to "Apps and Features" and search for "webview2 runtime" in the list to get the version:
hmurty
commented
3 years ago Doing this work on two PCs. Both have a date of 6/5/2021 for Microsoft.Edge.WebView2.Runtime.
vbryh-msft
commented
3 years ago Have tried to reproduce the issue without success. Code JS:
<div id="yourDiv">The content to refresh/reload</div>
<script>
function sleep(milliseconds) {
const date = Date.now();
let currentDate = null;
do {
currentDate = Date.now();
} while (currentDate - date < milliseconds);
}
function reload() {
var container = document.getElementById("yourDiv");
var today = new Date();
container.innerHTML = today.getHours() + ":" + today.getSeconds();
// Next line can be on with different values or off
// sleep(1000);
console.log("Refreshed");
};
</script>
.NET
for (int i = 0; i < 100; i++)
{
TimerTask(i);
await Task.Delay(2000);
}
private async void TimerTask(int index)
{
Debug.WriteLine("TimerTask checking for backFromJSScriptAsyncGSQRefresh: " + backFromJSScriptAsyncGSQRefresh + " index: " + index);
if (backFromJSScriptAsyncGSQRefresh == true)
{
Debug.WriteLine("ExecuteScriptAsync");
backFromJSScriptAsyncGSQRefresh = false;
await webView.ExecuteScriptAsync("reload();");
backFromJSScriptAsyncGSQRefresh = true;
}
}@AnQueth @hmurty - do you have any suggestions how to tweak this code to see ExecuteScriptAsync blocked? Could you please help with debugging the issue on your side with next few experiments:
"console.log("some log");" for your script in ExecuteScript and see if issue is still reproducible? Does script content impacts on the blocking or it is timing of script execution? hmurty
commented
3 years ago @vbryh-msft thank you for taking the time to do this test. I will try to get to your suggestions 1-3 during the coming week. If I understand your test correctly, the JavaScript that you run with ExecuteScriptAsync does not lead to execution of embedded JavaScript. The calculation of the time is done in the browser and does not lead to an HTTP call if I understand it correctly. In my application there is JavaScript execution as a part of ExecuteScruptAsync but also with the click the embedded JavaScript goes to the server and fetches the most recent data. So there are two pieces of JavaScript executing: a) one injected by the ExecuteScriptAsync and the other embedded on the page which leads to an HTTP SPA call. Thanks!
vbryh-msft
commented
3 years ago @hmurty Thanks for your suggestion. Used XMLHttpRequest with next code and .NET part with ExecuteScriptAsync("reload();") in for loop from previous test:
<button id ="demo_button" type="button" onclick="loadTimer()">Change Content</button>
<div id="demo">
<h2>Current time:</h2>
</div>
<script>
function loadTimer() {
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function () {
if (this.readyState == 4 && this.status == 200) {
document.getElementById("demo").innerHTML = this.responseText;
}
};
xhttp.open("GET", "https://appassets.example/timer.txt", true);
xhttp.send();
}
function reload() {
document.getElementById("demo_button").click();
}
</script>Requested timer.txt file is updated separately every second with current time. No luck with repro so far. Will wait for your findings. Also please let me know if you think this test could be updated more to closer simulate what is happening on your side, when you see the issue.
hmurty
commented
3 years ago @vbryh-msft Thanks for doing this. My code does pretty much the same thing. Next will try to implement your suggestions. Will set aside a day during this coming weekend.
vbryh-msft
commented
3 years ago Hi @AnQueth @hmurty could you please share your application where you see the issue? Thanks.
vbryh-msft
commented
3 years ago The issue should be fixed in Edge Canary 93.0.950.0+. Please let me know if you still able to repro it.
gribunin
commented
3 years ago The issue should be fixed in Edge Canary 93.0.950.0+. Please let me know if you still able to repro it.
Is it fixed in WebView2 runtime? The latest version of WebView2 runtime is 92.0.902.78 and I still having this problem there.
The code is very simple -- several times per hour the webview component navigates different pages and I am getting the page DOM by calling ExecuteScriptAsync:
private void webView22_NavigationCompleted(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationCompletedEventArgs e)
{
_ = ProcessBodyPageAsync();
}
private async Task ProcessBodyPageAsync()
{
string html = await webView22.ExecuteScriptAsync("document.documentElement.outerHTML");
// ......
}randomly after 10-20 hours of working ExecuteScriptAsync blocks and the code never executes past the line with ExecuteScriptAsync. The webview itself stays responsible, is able to do further navigations, but all new calls to ExecuteScriptAsync are blocked, the only way to make it working is to restart the app.
vbryh-msft
commented
3 years ago @gribunin this fix is not available in release yet. You would need to download a Canary channel to test it. Please let me know if you have more questions.
wgtgithub
commented
2 years ago here's the whole code. it's a messy mess because it's just hacky thing. the jist of it is login by using the form and submitting so the tokens google work. when the webpacked file is downloaded, intercept it and expose jquery globably. then manipulate the combo boxes which use jqueryui select2 plugin. when ajax request completes, run my selection code for the drop downs. then submit the form and start paging the results if it has pages then repeat everything but login when out of pages.
i did have this before, and it would block to. there is nothing there should block.
string selectOption = "document.documentElement.outerHTML"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); var html = Regex.Unescape(r);
`
using Azure.Messaging.ServiceBus; using Microsoft.Web.WebView2.Core; using Newtonsoft.Json; using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; using System.Net; using System.Net.Http; using System.Text; using System.Text.Json; using System.Text.RegularExpressions; using System.Threading; using System.Threading.Tasks; using System.Web; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; namespace autoastat { /// <summary> /// Interaction logic for MainWindow.xaml /// </summary> public partial class MainWindow : Window { ServiceBusClient serviceBusClient = new ServiceBusClient("E"); private const string Dir = "f:\\scrape"; private int _currentMake = 1; private int _currentModel = 1; private string _currentUri = ""; private bool _loggedin; private Uri _referer; private string _username = "login"; private string _password = "7Fpassword"; private ServiceBusSender _sbc; private int CMake { get => _currentMake; set { _currentMake = value; Save(); } } private int CModel { get => _currentModel; set { _currentModel = value; Save(); } } private void Save() { var ob = new { CModel = _currentModel, CMake = _currentMake, Page = _currentUri }; var d = JsonConvert.SerializeObject(ob); File.WriteAllText("data.json", d); } private void Load() { if (File.Exists("data.json")) { var d = JsonConvert.DeserializeObject<dynamic>(File.ReadAllText("data.json")); _currentMake = (int)d["CMake"].Value; _currentModel = (int)d["CModel"].Value; _currentUri = d["Page"].Value; } } public MainWindow() { AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException; InitializeComponent(); } private void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e) { } private async void Grid_Loaded(object sender, RoutedEventArgs e) { _sbc = serviceBusClient.CreateSender("scrape"); Load(); var env = await CoreWebView2Environment.CreateAsync(); await web.EnsureCoreWebView2Async(env); web.CoreWebView2.ProcessFailed += CoreWebView2_ProcessFailed; web.CoreWebView2.OpenDevToolsWindow(); //web.CoreWebView2.AddWebResourceRequestedFilter("https://autoastat.com/build/vehicle/main.1e19bf9f.js", // Microsoft.Web.WebView2.Core.CoreWebView2WebResourceContext.All); web.CoreWebView2.AddWebResourceRequestedFilter("*", Microsoft.Web.WebView2.Core.CoreWebView2WebResourceContext.All); web.CoreWebView2.WebResourceRequested += this.CoreWebView2_WebResourceRequested; web.CoreWebView2.DOMContentLoaded += this.CoreWebView2_DOMContentLoaded; web.CoreWebView2.NavigationStarting += this.CoreWebView2_NavigationStarting; web.CoreWebView2.WebResourceResponseReceived += this.CoreWebView2_WebResourceResponseReceived; web.CoreWebView2.NavigationCompleted += this.CoreWebView2_NavigationCompleted; web.CoreWebView2.Navigate("https://autoastat.com/en/login"); } private void CoreWebView2_ProcessFailed(object sender, Microsoft.Web.WebView2.Core.CoreWebView2ProcessFailedEventArgs e) { } private void CoreWebView2_NavigationStarting(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationStartingEventArgs e) { } private async void CoreWebView2_WebResourceRequested(object sender, Microsoft.Web.WebView2.Core.CoreWebView2WebResourceRequestedEventArgs e) { if (_referer != null) e.Request.Headers.SetHeader("Referer", _referer.ToString()); if (e.Request.Uri.Contains("main") && e.Request.Uri.Contains("js")) { HttpClient cl = new HttpClient(); var d = e.GetDeferral(); var data = await cl.GetStringAsync(e.Request.Uri); const string s = "(function(e){n(\"8h6d\");"; data = data.Insert(data.IndexOf(s) + s.Length, "window.$=e;"); Stream st = new MemoryStream(Encoding.UTF8.GetBytes(data)); e.Response = web.CoreWebView2.Environment.CreateWebResourceResponse(st, 200, "OK", $"Content-Type: application/javascript"); d.Complete(); } } private async void CoreWebView2_DOMContentLoaded(object sender, Microsoft.Web.WebView2.Core.CoreWebView2DOMContentLoadedEventArgs e) { if (web.Source.PathAndQuery.EndsWith("en/")) { } } private async void CoreWebView2_WebResourceResponseReceived(object sender, Microsoft.Web.WebView2.Core.CoreWebView2WebResourceResponseReceivedEventArgs e) { if (e.Request.Uri.Contains("en/form?")) { try { var s = await e.Response.GetContentAsync(); var buf = new byte[s.Length]; s.Read(buf, 0, buf.Length); var html = Encoding.UTF8.GetString(buf); HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument(); doc.LoadHtml(html); if (!Directory.Exists(Dir)) Directory.CreateDirectory(Dir); foreach (var node in doc.DocumentNode.SelectNodes("//div[@class='block']")) { //await File.WriteAllTextAsync( // $"{Dir}\\{CMake}_{CModel}_{DateTimeOffset.UtcNow.Ticks}", node.OuterHtml); var a = node.SelectSingleNode(".//a").GetAttributeValue("href", ""); var al = "https://autoastat.com" + a; var fn = GetFileName(al); await _sbc.SendMessageAsync(new ServiceBusMessage(al)); using (var fs = File.Open(Dir + "\\" + fn, FileMode.Append, FileAccess.Write, FileShare.ReadWrite)) { using (var sr = new StreamWriter(fs)) { sr.WriteLine(al); } } //File.AppendAllText($"{Dir}\\{fn}", al + "\r\n"); } HtmlAgilityPack.HtmlNode? paging = null; var pn = doc.DocumentNode.SelectNodes("//a[@class='page-link']"); if (pn != null) { paging = pn.LastOrDefault(); } if (paging == null) { await Task.Delay(new Random().Next(60000, 120000)); _currentUri = ""; Save(); CModel++; _referer = null; web.Source = new Uri("https://autoastat.com/en/"); } else { _referer = web.Source; string u = "https://autoastat.com" + paging.GetAttributeValue("href", "").Replace("&", "&"); var src = HttpUtility.ParseQueryString(_referer.Query).Get("page"); var ns = HttpUtility.ParseQueryString(new Uri(u).Query).Get("page"); if (!int.TryParse(src, out var old)) { old = 0; } if (!int.TryParse(ns, out var news)) { } if (news > old) { await Task.Delay(new Random().Next(60000, 120000)); web.Source = new Uri(u); _currentUri = u; Save(); } else { await Task.Delay(new Random().Next(60000, 120000)); CModel++; _referer = null; _currentUri = ""; Save(); web.Source = new Uri("https://autoastat.com/en/"); } } } catch (Exception ex) { Debug.WriteLine(ex.ToString()); web.Reload(); } } if (e.Request.Uri.Contains("main") && e.Request.Uri.EndsWith(".js")) { //var s = await e.Response.GetContentAsync(); //var buf = new byte[s.Length]; //s.Read(buf, 0, buf.Length); //string g = Encoding.UTF8.GetString(buf); } if (e.Request.Uri.Contains("/en/make")) { try { var s = await e.Response.GetContentAsync(); // _makes = await JsonSerializer.DeserializeAsync<Make>(s); var buf = new byte[s.Length]; s.Read(buf, 0, buf.Length); var _makes = JsonConvert.DeserializeObject<Make>(Encoding.UTF8.GetString(buf)); Task.Delay(1000).ContinueWith(async (t) => { Application.Current.Dispatcher.Invoke(async () => { string el = "select-make"; string selectOption = " try{ var el = document.getElementById(\"" + el + "\");" + "var op = el.options[" + CMake + "]; " + "$('#" + el + "').val(op.value);" + "$('#" + el + "').trigger('change');" + "} catch(e){alert(e);}"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); }); }); } catch (Exception ex) { Debug.WriteLine(ex.ToString()); } } if (e.Request.Uri.Contains("en/model")) { Task.Delay(500).ContinueWith(async (t) => { Application.Current.Dispatcher.Invoke(async () => { string el = "select-model"; string selectOption = " function modelChange(){ try{ var el = document.getElementById(\"" + el + "\");" + "var op = el.options[" + CModel + "]; " + "$('#" + el + "').val(op.value);" + "$('#" + el + "').trigger('change');" + "return 'good';" + "} catch(e){ return 'bad'; }}\r\n" + "modelChange();"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); if (r == "\"bad\"") { CMake++; CModel = 1; web.Reload(); //web.Source = new Uri("https://autoastat.com/en/"); } }); }); } if (e.Request.Uri.Contains("en/date")) { var s = await e.Response.GetContentAsync(); // _makes = await JsonSerializer.DeserializeAsync<Make>(s); var buf = new byte[s.Length]; s.Read(buf, 0, buf.Length); string ds = ""; try { ds = Encoding.UTF8.GetString(buf); var dates = JsonConvert.DeserializeObject<Dates>(ds); var dts = dates.results.Select(z => DateTime.Parse(z)).Where(z => z >= DateTime.Today.AddDays(-14)).OrderByDescending(p => p).ToArray(); if (dts.Count() == 0) { dts = dates.results.Select(z => DateTime.Parse(z)).OrderByDescending(p => p).ToArray(); } var old = dts.LastOrDefault(); var n = dts.FirstOrDefault(); if (old == default) old = n; if (old == default && n == default) { return; } Task.Delay(20000).ContinueWith(async (t) => { Application.Current.Dispatcher.Invoke(async () => { string selectOption = " try{ $('#select-date-from').val('" + old.ToString("yyyy-MM-dd") + "') ; " + "$('#select-date-to').val('" + n.ToString("yyyy-MM-dd") + "') ; " + "$('#search').click(); " + "} catch(e){console.log(e);}"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); }); }); } catch { Debug.WriteLine(ds); CModel++; web.Reload(); } } } private async void CoreWebView2_NavigationCompleted(object sender, Microsoft.Web.WebView2.Core.CoreWebView2NavigationCompletedEventArgs e) { if (_loggedin) { _loggedin = false; if (_currentUri != string.Empty) { web.Source = new Uri(_currentUri); return; } else { _currentUri = web.Source.ToString(); Save(); } } if (web.Source.PathAndQuery.EndsWith("login")) { string login = "document.getElementById('_username').value = '" + _username + "';" + "document.getElementById('_password').value = '" + _password + "';" + " document.querySelectorAll(\"button[type='submit']\")[2].click();"; await web.CoreWebView2.ExecuteScriptAsync(login); _loggedin = true; } } private string GetFileName(string al) { int c = 0; foreach (byte x in al) { c += x; } int r = c % 1; return r.ToString() + ".urls"; } } }
long long long coding....
CenturionYun
commented
2 years ago @gribunin this fix is not available in release yet. You would need to download a Canary channel to test it. Please let me know if you have more questions.
@vbryh-msft When can we expect this fix be in the release version? We also face this problem, hope it can be fixed soon.
vbryh-msft
commented
2 years ago @CenturionYun the issue was fixed and available in releases since 93.0.961.0 version.
CenturionYun
commented
2 years ago @CenturionYun the issue was fixed and available in releases since 93.0.961.0 version.
@vbryh-msft Does it require a Runtime fix or SDK fix? I read the SDK release note from here https://docs.microsoft.com/en-us/microsoft-edge/webview2/release-notes, is says it had been fixed in 1.0.955-prerelease, but no prod release mentions it. We still see this issue on our users' side with latest Runtime and SDK, but we cannot reproduce it.
vbryh-msft
commented
2 years ago @CenturionYun - as per our release process all bugfixes from the prerelease are available in the next release also(we will see if we can make it more clear in the notes). Also I have checked - the fix for this issues requires Runtime update. What exactly you see on your user's side?
CenturionYun
commented
2 years ago @CenturionYun - as per our release process all bugfixes from the prerelease are available in the next release also(we will see if we can make it more clear in the notes). Also I have checked - the fix for this issues requires Runtime update. What exactly you see on your user's side?
@vbryh-msft We call ExecuteScript in an Win32 App to get time(just for test) every 2 seconds, we find that it does not return after a few calls, it only happen on some computers, not all. BTW, will WebView2 auto sleep or freeze itself like Edge when app becomes inactive?
vbryh-msft
commented
2 years ago @CenturionYun - WebView2 will not auto sleep by itself when app is inactive - you can do it using TrySuspend API About ExecuteScript not returning - do you know for how long the app is running and how many requests are send before facing the issue? Which runtime are you using? Any repro suggestion could help to identify what is causing the problem.
this script and other scripts have a tendency to block forever on the await for the script to finish. When this happens the webview2 is responsive in the ui, but the ExecuteScruptAsync will break again after refreshing it.
string selectOption = "document.documentElement.outerHTML"; var r = await web.CoreWebView2.ExecuteScriptAsync(selectOption); var html = Regex.Unescape(r);
Version SDK: latest nuget Runtime: i have prod and dev builds installed for edge and the webview2 runtime Framework: wpf OS: win 10 20h2
AB#33665218