Closed rkfg closed 2 months ago
benma
commented
2 months ago Hi
So it works in 4.41 but it is a regression in 4.42? :thinking:
Could you try running this:
QTWEBENGINE_REMOTE_DEBUGGING=23456 BitBoxThen opening http://127.0.0.1:23456/, clicking BitBoxApp, then Console, and copying all the output there? Maybe there is a relevant error message there.
rkfg
commented
2 months ago Yes, it's a regression. The console says a very interesting thing: Failed to load module script: The server responded with a non-JavaScript MIME type of "text/x-matlab". Strict MIME type checking is enforced for module scripts per HTML spec. The relevant file/line is index-cNUc17VX.js:1 so it seems that the main file didn't load because of it.
benma
commented
2 months ago x-matlab, that is quite mysterious.
It is not quite clear to me if the index JS file could not be loaded, or if the index.js file tried to load something which failed. The index.js file is loaded as a local resource by QtWebEngine, that should be no problem :thinking:
Can you check the Network tab in the developer console? Is there any resource shown there?
rkfg
commented
2 months ago
I reloaded the devtools and it should've caused the UI reload so here's what it says. The UI doesn't load obviously but at least there's this log.
benma
commented
2 months ago How about the output of:
cd /opt/bitbox
ldd ./BitBox | grep -i webengine libQt5WebEngineWidgets.so.5 => /opt/bitbox/./lib/libQt5WebEngineWidgets.so.5 (0x00007fcdf967d000)
libQt5WebEngineCore.so.5 => /opt/bitbox/./lib/libQt5WebEngineCore.so.5 (0x00007fcdf0200000)Note that the problem exists in the .AppImage too which should be not dependent on my system.
benma
commented
2 months ago When I reload the dev tools (after a working UI), I get:
and the app shows "Your file was not found".
It is strange that it behaves differently for us, seeing that the WebEngine libraries are bundled and the asset is loaded statically. This is why I had thought that maybe your app may have accidentally linked to another lib.
What if you tried this anyway just to rule it out:
LD_LIBRARY_PATH=/opt/bitbox/lib BitBoxor ran the BitBox from the local folder:
cd /opt/bitbox
./BitBox?
rkfg
commented
2 months ago Both commands have the same result, the window is empty. Just to make sure, when I open the devtools link I see this:
I click this line and the devtools open. After Ctrl+R here, I see this:
The contents of the JS file look legit (i.e. it's not a wrong file being loaded which might cause this incorrect mime type):
benma
commented
2 months ago For me it shows the correct mimetype text/javascript.
Browsing commits since 4.41.0, the only potentially relevant commits could be
https://github.com/digitalbitbox/bitbox-wallet-app/commit/db555d7d86af893987f48e1ab9e555692becc5b3
and
https://github.com/digitalbitbox/bitbox-wallet-app/commit/b6749d4aee6d061882b45dc86e20d1a96c81dcd8
Though I don't really see how they could change the mimetype - I thought the mimetype is determined by Chromium by the .js extension. And we use the same bundled version, so it should behave the same anyway.
Are you able to build the app yourself? If not, I can try to make builds for your reverting these commits to see if they caused it.
Maybe you could also try running it in a VirtualBox VM or Ubuntu Liveboot, just to see if there is something on your system that could be interfering.
rkfg
commented
2 months ago It works in a Debian VM, the system there is quite outdated so I'll update it and see if it breaks. There are not many mentions of that mime type in my system but maybe something here rings a bell? I'm not very familiar with Qt Web engine. I also never used matlab and I don't have it installed.
rkfg
commented
2 months ago Still works after update. I'll try to build the app later.
benma
commented
2 months ago I also don't have matlab/octave installed and have similar results in the /usr/share folder.
rkfg
commented
2 months ago I initialized a docker environment and am currently bisecting it testing appimages. The master version is broken, 4.41.0 works.
rkfg
commented
2 months ago Bisect says it's b6749d4aee6d061882b45dc86e20d1a96c81dcd8 as you assumed. So what now?
rkfg
commented
2 months ago I manually reverted it on top of master but it didn't help. There were many conflicts in the process and resolving the package-lock.json conflicts is hard. Not sure I did it right. However, I can confirm that b6749d4aee6d061882b45dc86e20d1a96c81dcd8 is broken and bb5a488f3497fece41b2541cb3780d4e89fd9b87 (the immediate parent) is good.
thisconnect
commented
2 months ago I believe the relevant dependency in https://github.com/digitalbitbox/bitbox-wallet-app/commit/b6749d4aee6d061882b45dc86e20d1a96c81dcd8 is vite all others are only for types, linting and testing.
I'll try to make a quick test and compare the unminified build (make buildweb only) of the bundle.js, maybe I see something noticable that changed..
There were many conflicts in the process and resolving the package-lock.json conflicts is hard.
In this case you could just delete and recreate the package-lock.json by rm -rf node_modules && rm package-lock.json && npm install
rkfg
commented
2 months ago In this case you could just delete and recreate the package-lock.json by rm -rf node_modules && rm package-lock.json && npm install
I think it wouldn't do what's expected (I just tried and nothing changed). Many dependencies are specified as a caret range ^a.b.c which would select the latest version that has the major version a today. It might match what was selected back then but I wouldn't bet on it. I already had plenty of situations where some breakage occurred between the minor updates. For example, Yarn+Vite+PnP+Typescript is an interesting bunch, the TS version is specified as ^5.2.2 by default and it selects 5.4.4 which runs fine in development but absolutely breaks during build (https://github.com/yarnpkg/berry/issues/5814#issuecomment-2047449149). So if I had 5.3.3 (which works) in the lock file and removed it, it'd select 5.4.4 (which is broken).
thisconnect
commented
2 months ago attached is a gist with the diff of JS and HTML file of the 2 unminified versions, using make buildweb with this vite.config.js
diff --git a/frontends/web/vite.config.mjs b/frontends/web/vite.config.mjs
index b012fd3a3..ea2a3b3b7 100644
--- a/frontends/web/vite.config.mjs
+++ b/frontends/web/vite.config.mjs
@@ -11,6 +11,14 @@ export default defineConfig(() => {
build: {
modulePreload: false,
outDir: 'build',
+ minify: false,
+ rollupOptions: {
+ output: {
+ entryFileNames: 'assets/[name].js',
+ chunkFileNames: 'assets/[name].js',
+ assetFileNames: 'assets/[name].[ext]',
+ }
+ }
},
plugins: [
react(),https://gist.github.com/thisconnect/ceea8d1935c126a66447bb7fc0289ac4
rkfg
commented
2 months ago As an experiment, I removed text/x-matlab definitions from /usr/share/mime/packages/freedesktop.org.xml and then ran update-mime-database /usr/share/mime. Interestingly enough, it now detects a different mime type (and still doesn't work):
The modelica definitions include these matchers:
<magic priority="10">
<match type="string" value="//" offset="0"/>
</magic>
<magic>
<match type="string" value="function" offset="0"/>
</magic>
<magic>
<match type="string" value="class" offset="0"/>
</magic>
<magic>
<match type="string" value="model" offset="0"/>
</magic>
<magic>
<match type="string" value="record" offset="0"/>
</magic>And since that index.js starts with function it works. The matlab type also matches because of this:
<magic priority="10">
<match type="string" value="%" offset="0"/>
</magic>
<magic>
<match type="string" value="function" offset="0"/>
</magic>
<glob pattern="*.m"/>Now the question is why it doesn't break the earlier version and who's responsible for setting the Content-Type header field.
rkfg
commented
2 months ago Probably the problem is this: https://bugs.archlinux.org/task/80279 Patch here: https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3714
Still not sure why it works fine in the VM with the same package version. Maybe something else is triggering the bug?
rkfg
commented
2 months ago After minification, the working file assets/index-dc2dc9e6.js starts with var QS=Object.defineProperty;var ZS=(t,e,n)=>e in t?QS
The broken file assets/index-LLzg5e7T.js starts with:
function __vite__mapDeps(indexes) {
if (!__vite__mapDeps.viteFileDeps) {
__vite__mapDeps.viteFileDeps = []
}
return indexes.map((i) => __vite__mapDeps.viteFileDeps[i])
}The word function in the very beginning of the file triggers the magic match for text/x-matlab and other such file types. Since the working file starts with var it's not triggered and it works. I'm unsure who's to blame because xdg-mime query filetype assets/index-LLzg5e7T.js returns text/javascript. It's like for Qt the magic bytes are more important than the file extension.
benma
commented
2 months ago Wow, great debugging! :star: And what a crazy and unlikely bug too :exploding_head: I quickly tried browsing the qtwebengine source to see how the mimetype is determined, but I couldn't figure it out.
Still not sure why it works fine in the VM with the same package version. Maybe something else is triggering the bug?
Maybe you can find the difference in the mime files between the two systems, and maybe there are some commands that can list packages that touch these files that you could run.
In any case, now we need to decide what to do about this. Options are:
Any other ideas?
rkfg
commented
2 months ago Maybe you can find the difference in the mime files between the two systems, and maybe there are some commands that can list packages that touch these files that you could run.
Both systems now run Debian testing and the package versions are identical, at least shared-mime-info (2.4-1). There's certainly something shady going on but I'm not sure where to dig. I straced the binary but there's not much info, it loads the usual mime caches as well as my user's one in ~/.local but there's nothing special there. I also ran the program under different $HOME but on the same machine and the behavior is the same, so it's not my user files but something in the system. Another attempt was using LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu to load the system Qt libs, unfortunately the program segfaults in this case without even showing the window.
I'd like to try a Qt update, not sure how to do it properly for docker build. I changed the Qt version to 5.15.17 in frontends/qt/docker-qt5base/Dockerfile and also added gem install dotenv -v 2.8.1 in scripts/docker_install.sh because it complained about it. Then I did make dockerinit and make dockerdev but in the final docker container the Qt version is still 5.15.2.
We don't know if this bug is rare, we'll see in the next few days because 4.42.0 was just released, maybe not many people updated yet. I'd really like to know how QWebEngineView assigns mime types, I suppose that's where it happens from the code. After all, QWE works as both client and server since it handles the qrc:/ URLs and returns the data. Chromium certainly can't handle Qt's binary resources and most probably the entire response, including headers, is produced by Qt. If that part of the code can be isolated I'd love to run any tests to see why it chooses these mime types. Maybe it uses some system library for that with a cryptic name, maybe it even loads it dynamically so we don't see it in ldd output.
benma
commented
2 months ago @rkfg
I haven't verified but it seems likely that this is the code that identifies the mimetype:
Which would be this method:
https://doc.qt.io/qt-5/qmimedatabase.html#mimeTypeForFile
With the default match mode:
QMimeDatabase::MatchDefault 0x0 Both the file name and content are used to look for a match
In the Qt codebase, this would be this line:
https://github.com/qt/qtbase/blob/v5.15.2/src/corelib/mimetypes/qmimedatabase.cpp#L577
which calls this function?
https://github.com/qt/qtbase/blob/v5.15.2/src/corelib/mimetypes/qmimedatabase.cpp#L355
which only looks at the content if the extension yields multiple candidates.
A newer Qt version has different code, and it seems like only the filename is considered edit: reading the cod again it seems to behave the same, but can't be sure unless we test it
https://github.com/qt/qtbase/blob/v6.7.0/src/corelib/mimetypes/qmimedatabase.cpp#L492
Again, I was just browsing code. To verify, you'd have to call QMimeDatabase::mimeTypeForFile() (using the QFileInfo argument, not the filename argument) in a demo project or in the app's main.cpp, sth like:
QMimeDatabase mimeDatabase;
// make fileInfo of type QFileInfo
auto mimeType = mimeDatabase.mimeTypeForFile(&fileInfo);
std::cout << "MimeType: " << mimeType.name().toStdString() << std::endl;@rkfg could you build and test this branch to see if it solves your problem? It forces mime type resolution by extension only:
https://github.com/benma/bitbox-wallet-app/tree/mimetype/
As predicted above, it is quite a bit of weird code to accomplish this, but it's worth a try.
rkfg
commented
2 months ago Nice, this version with mime patch works well! Although it's clearly a hacky solution, I'd rather find the real reason. Thanks for the pointers ( :wink: ), I'll take a look and experiment with debug prints.
Tried this:
#include <QCoreApplication>
#include <QFile>
#include <QFileInfo>
#include <QMimeDatabase>
#include <QDebug>
int main(int argc, char *argv[])
{
QFile file("index-cNUc17VX.js");
QFileInfo fileInfo(file);
file.open(QFile::ReadOnly);
QMimeDatabase mimeDatabase;
QMimeType mimeType = mimeDatabase.mimeTypeForData(file.readAll());
qInfo() << mimeType.name();
return 0;
}Using the file I saved from the app. It returns "text/x-matlab" on both machines while QMimeType mimeType = mimeDatabase.mimeTypeForFile(file); (and fileInfo) returns "application/javascript" in docker and "text/javascript" on host.
rkfg
commented
2 months ago which only looks at the content if the extension yields multiple candidates.
You might be onto something here. Too bad there's absolutely no debug output in that function so it's not possible to find if it found more than one candidate by extension.
Oh wtf, I solved it :joy: This is indeed what caused it, I grepped /usr/share/mime by '\*\.js' and found the file packages/monodevelop.xml that also has *.js glob in it. Here's the file:
<?xml version="1.0" encoding="UTF-8"?>
<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info">
<mime-type type="text/x-csharp">
<comment xml:lang="en">C# source</comment>
<glob pattern="*.cs"/>
</mime-type>
<mime-type type="text/x-msil">
<comment xml:lang="en">IL source</comment>
<glob pattern="*.il"/>
</mime-type>
<mime-type type="text/x-nemerle">
<comment xml:lang="en">Nemerle source</comment>
<glob pattern="*.n"/>
</mime-type>
<mime-type type="text/x-boo">
<comment xml:lang="en">Boo source</comment>
<glob pattern="*.boo"/>
</mime-type>
<mime-type type="text/x-vb">
<comment xml:lang="en">VB source</comment>
<glob pattern="*.vb"/>
</mime-type>
<mime-type type="text/x-js">
<comment xml:lang="en">JScript source</comment>
<glob pattern="*.js"/>
</mime-type>
<mime-type type="application/x-aspx">
<comment xml:lang="en">ASP.NET page</comment>
<glob pattern="*.aspx"/>
</mime-type>
<mime-type type="application/x-ashx">
<comment xml:lang="en">ASP.NET handler</comment>
<glob pattern="*.ashx"/>
</mime-type>
<mime-type type="application/x-ascx">
<comment xml:lang="en">ASP.NET user control</comment>
<glob pattern="*.ascx"/>
</mime-type>
<mime-type type="application/x-asix">
<comment xml:lang="en">ASP.NET image generator</comment>
<glob pattern="*.asix"/>
</mime-type>
<mime-type type="application/x-axd">
<comment xml:lang="en">ASP.NET handler</comment>
<glob pattern="*.axd"/>
</mime-type>
<mime-type type="application/x-web-config">
<comment xml:lang="en">ASP.NET web config</comment>
<glob pattern="web.config"/>
</mime-type>
<mime-type type="application/x-machine-config">
<comment xml:lang="en">Mono machine config</comment>
<glob pattern="machine.config"/>
</mime-type>
<mime-type type="application/x-config">
<comment xml:lang="en">Mono application config</comment>
<glob pattern="*.config"/>
</mime-type>
<mime-type type="application/x-master-page">
<comment xml:lang="en">ASP.NET master page</comment>
<glob pattern="*.master"/>
</mime-type>
<mime-type type="application/x-resources">
<comment xml:lang="en">resources bundle</comment>
<glob pattern="*.resources"/>
</mime-type>
<mime-type type="application/x-resourcesx">
<comment xml:lang="en">resx bundle</comment>
<glob pattern="*.resx"/>
</mime-type>
<mime-type type="application/x-remoting">
<comment xml:lang="en">remoting handler</comment>
<glob pattern="*.rem"/>
</mime-type>
<mime-type type="application/x-soap-remoting">
<comment xml:lang="en">soap remoting handler</comment>
<glob pattern="*.soap"/>
</mime-type>
<mime-type type="application/x-asmx">
<comment xml:lang="en">ASP.NET web service</comment>
<glob pattern="*.asmx"/>
</mime-type>
<mime-type type="application/x-prjx">
<comment xml:lang="en">SharpDevelop project</comment>
<glob pattern="*.prjx"/>
</mime-type>
<mime-type type="application/x-cmbx">
<comment xml:lang="en">SharpDevelop solution</comment>
<glob pattern="*.cmbx"/>
</mime-type>
<mime-type type="application/x-mdsx">
<comment xml:lang="en">SharpDevelop solution extension</comment>
<glob pattern="*.mdsx"/>
</mime-type>
<mime-type type="application/x-mdp">
<sub-class-of type="text/plain"/>
<comment xml:lang="en">MonoDevelop project</comment>
<glob pattern="*.mdp"/>
</mime-type>
<mime-type type="application/x-mds">
<sub-class-of type="text/plain"/>
<comment xml:lang="en">MonoDevelop solution</comment>
<glob pattern="*.mds"/>
</mime-type>
<mime-type type="application/x-csproj">
<sub-class-of type="text/plain"/>
<comment xml:lang="en">Visual Studio .NET C# project</comment>
<glob pattern="*.csproj"/>
</mime-type>
<mime-type type="application/x-vbproj">
<sub-class-of type="text/plain"/>
<comment xml:lang="en">Visual Studio .NET VB.NET project</comment>
<glob pattern="*.vbproj"/>
</mime-type>
<mime-type type="application/x-sln">
<sub-class-of type="text/plain"/>
<comment xml:lang="en">Visual Studio .NET Solution</comment>
<glob pattern="*.sln"/>
</mime-type>
<mime-type type="application/x-disco">
<comment xml:lang="en">Static Discovery File</comment>
<glob pattern="*.disco"/>
</mime-type>
<mime-type type="application/x-asax">
<comment xml:lang="en">ASP.NET Global Application Class</comment>
<glob pattern="*.asax"/>
</mime-type>
<mime-type type="application/x-wsdl">
<comment xml:lang="en">Webservices Description File</comment>
<glob pattern="*.wsdl"/>
</mime-type>
</mime-info>It belongs to the monodevelop package. So to reproduce either try installing that package or create a file /usr/share/mime/packages/monodevelop.xml with the contents from above, then do update-mime-database /usr/share/mime and after that BitBox breaks. This bug is wild!
My monodevelop version is 5.10.0.871-2, it seems to be a very old version from 2015. However, I downloaded a .deb for the last version they have (from 2019) and it also has the same xml file with the same contents, at least the *.js part is present.
benma
commented
2 months ago QMimeType mimeType = mimeDatabase.mimeTypeForData(file.readAll());This is the wrong function afaik, nvm I saw now that you actually tried the right function toomimeTypeForFileNameAndData is the one used, which also takes into account the name/extension, not just the content.
Anyway, great job finding the culprit!
Now I still don't know whether we should pursue the above patch or leave it as it is :grin:
rkfg
commented
2 months ago Yeah, that's a tricky question. There are many parties to blame for the bug but it's pretty niche since monodevelop is probably abandoned for good today. I have no idea how I got it installed in the first place, I never developed for .NET. Maybe it was pulled as a dependency even though it doesn't exist in the current Debian package archives.
So I think leaving everything as is would be good and if it happens to someone else we can simply link them this issue. Thanks for you help!
Only a dark empty window is shown but nothing in it. There are no errors in the log either. Might be the same issue as #2618, I tried both .deb and .AppImage, same symptoms. The app itself seems to work as there's a password prompt on my BitBox02, and when I enter it the device unlocks, there's also some activity in the log after that as well. I downgraded the app (.deb) to 4.41.0 and it works fine again.
I use Debian testing, NVIDIA RTX 3090 Ti if hardware is related. There was a recent Chromium bug in Steam that caused similar issues, the main window is empty. Turning hardware acceleration off helps but it seems this program already uses software rendering. I tried setting
BITBOXAPP_RENDER=autoas said in the docs but it didn't help.Part of the log: