till213 / SkyDolly

Sky Dolly connects with Flight Simulator 2020 and records the flight path and basic instruments for replay.
MIT License
77 stars 9 forks source link

skydolly.exe won't open #117

Closed Bloodrule closed 8 months ago

Bloodrule commented 11 months ago

I can't open skydolly.exe. I have unzipped 0.16.2 (and tried 0.16.1) several times but every time I try to open the executable nothing happens. No error messages and no sign of the executable in Windows task manager. Checksums are all OK.

If I connect to the same unzipped folder on my laptop, Sky Dolly opens normally.

My installation of MSFS runs normally in all respects. Windows 11 is up to date. Are there critical files needed for Sky Dolly other than those in the zip file?

Bloodrule commented 11 months ago

I got it to work by running skydolly.exe in Windows 8 compatibility mode. Very strange. Is this expected behaviour?

till213 commented 11 months ago

Hello Bloodrule,

Thank you for raising this issue! As you actually (presumably) bothered to read some release notes and about the checksums and you actually bothered to check your ZIP archive checksums against the published ones I assume you know a little bit or two about computers ;) While I try not to over-explain things I'll also try to give you a satisfactory explanation of what I think may go wrong. We might end up in some technical details, but again, I'll try to find some middle ground between of what I think you already (might) know about programming and what you might not understand at all. So let's start...

PART I - MISSING DLLs?

First off, the provided ZIP should be "self-contained", means: all required dependencies (DLLs) should be present, including some C++ runtime DLL (just for the curious: Sky Dolly is compiled using the GCC compiler from the MinGW suite, so the C++ runtime DLL would be libstdc++-6.dll - as an example).

The notable exception is the default Visual Studio C runtime (specifically: MSVCRT.DLL) that is expected to be already part of your Windows operating system. In fact, most applications (including MSFS itself) rely on this C runtime library, so chances are that even if that C runtime did not ship with Windows that it was already provided alongside some other application (I mention again MSFS itself).

Remark: I once had to completely re-install Windows 10 "from scratch" and one of the first things (before installing anything else, like another browser or so...) that I tried was to download Sky Dolly and run it. And it worked, because in fact the required C runtime was already part of the Windows 10 installation.

There is a (free) developer tool called "Dependency Walker" that exists as far back as Windows 95 (possibly even longer) that reliably shows all existing - and especially missing - dependencies of any given DLL and/or .exe. Most importantly it shows where it would pick them up when launching the corresponding DLL/.exe, from its current location.

For the mentioned GCC C++ runtime - the libstdc++-6.dll, I get on my system (as an example):

grafik

So in other words, on my Windows 10 system the Visual Studio C runtime is indeed located at:

C:\windows\system32\MSVCRT.DLL

But if it was missing it can easily be installed via the official Microsoft "C/C++ redistribution package" (again, most other applications bring this along with their own installer, so it is highly unlikely that this very basic C runtime DLL is actually missing):

https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

You'd most likely want the version for 64bit ("X64") CPUs:

https://aka.ms/vs/17/release/vc_redist.x64.exe

-> But before you go and install this package (which wouldn't hurt) please read on...

Assuming that such a dependency - any DLL, in fact - would really be missing then Windows would complain about such a missing dependeny with an error dialog.

Or in other words: especially since you already wrote that Sky Dolly seems to work in "Windows 8 compatibiliy mode" we can almost certainly rule out any missing dependency.

End of chapter I ;) I'll do a follow-up post with my latest suspicion of what may go wrong - and how you could help me track the root cause down!

P.S. If you paid very close attention to the above screenshot you may have noticed that it says in the DLL file path "v0.16.3" - don't worry, such a version wasn't published yet, so you didn't miss anything - I'll come to that v0.16.3 shortly, in a follow-up post...

till213 commented 11 months ago

CHAPTER II - AN OPEN SECRET

So yes, I confess: I "cheated" a bit when implementing Sky Dolly, specifically with regards to so-called "exceptions" - before I go into some technical details (and possibly "loose you" or "bore you to death", since you already know...), here's in summary what I expect:

I hope when logging / showing the type of the exception (and possibly where it "comes from") that this will reveal the true nature of this error! For now I suspect that this might be some "file / directory" permission issue). That is, Sky Dolly will try to create a "Sky Dolly" folder inside your default "Documents" folder (typically at C:\Users\Your User\Documents\"). It is well possible that Windows 11 allows to set some more fine-granular file access controls - and while the Qt toolkit (that I use for the user interface, but also to read/write files etc.) should not throw any exceptions on its own (it is supposed to be "exceptions free") it is well possible that my own code somehow stumbles over some non-initialised value.

Again, I hope the nature of the exception will reveal more...

I would greatly appreciate if you could install and try this "test build" and let me know about its outcome! I will let you know once it is available here on github.com (I won't publish it on flightsim.to - btw. all ZIP archives provided here on github.com and over at flightsim.to are identical).

till213 commented 11 months ago

And here are some technical details - but ignore this, this is just some background info.

In most programming languages you can do "error handling" in essentially two ways:

The advantage of "error codes" is that they are "simple to understand", but "tedious to implement correclty": the caller of some function (that returns possible error codes) may simply "ignore" such error codes (and the error may initially go undetected, causing havoc later on).

This disadvanage is supposed to be fixed by exceptions, as they can also be initially be ignored by the immediate caller of some function, but instead be called "way up in the call hierarchy", at some convenient place where it actually makes sense to do all (common) error handling.

Now in most languages (other than C++) this is very convenient, as most modern computer languages do have what is known as "(memory) garbage collector". That is, any memory that is not being referenced anymore by any "variable" in the program is automatically reclaimed by the garbage collector and hence available again - otherwise we'd have what is known as a "memory leak" (memory is "lost" in the sense that the application is consuming more and more RAM, as it keeps running).

WIth C++ there exists no garbage collection. And here comes the problem with exceptions: by design they "directly bail out of any function" at the very moment they are thrown. Any memory that has been allocated "on the heap" just before the exception occurs is hence potentially lost, as in C++ we always need to explicitly "delete objects" (= free memory) ourselves!

However! C++ has an answer that comes in two forms:

So essentialy: if we don't allocate memory on the heap ourselves, but instead let a special object - that crucially is allocated on the stack - allocate the memory for us, typically in its own constructor, then we get what is known as RAII (ok, here we go: "Resource Acquisition Is Initialisation" - I told you its a terrible, non-descriptive name ;)).

That "RAII" object will then also deallocate the previously allcoated memory (or generally speaking: the "resource": could also be some file handle, an open network socket or any other precious operating system resource that needs to be "closed" after use) in its own destructor.

Now comes the crucial part: if we allocate this RAII object on the stack (and again: not on the heap with a call to "new") then its constructor will be called in any case - no matter what! Especially also if an exception is thrown within the same function (or "scope").

And now we have everything together:

The good news: Sky Dolly is applying this RAII pattern to way over 90%, by using "smart pointers" (most often std::unique_ptr): those "smart pointers" follow exactly the RAII pattern.

Most other cases are explicit memory allocations not using RAII: this is when allocating QWidgets and passing them over to some parent QWidget: Qt has its own "parent/child" memory management, that is if some parent is deleted it will automatically delete all its children for us.

And then there are the remaining cases where "I am simply not sure" whether any memory leaks would occur when I'd start throwing exceptions in my own code (as I said, Qt itself does not (should not) throw any exceptions, according to its API specification, and while another 3rd-party library that I suse - the GeographicLib - does throw exceptions I am properly catching them (there are very few locations in the code where I am actually calling functionality from the GeographicLib.

So what did I do? I cheated! By explicitly declaring almost each and every method of my own code as "noexcept". That is, I am boldly declaring that none of my own methods would ever throw an exception!

That is not only assuming that I made no programming errors ever that would cause a possible exception (such as any "division by 0" and the like), but even more boldy that no "3rd-party functions" that I am calling are not throwing any exceptions - ever!

And the later point is technically wrong, by the C++ standard: because any memory allocation (no matter if done via RAII or a "new" on its own) may fail with an "out of memory excpetion".

Of course I furthermore boldly assume that anyone running MSFS has more than enough RAM and won't be using Sky Dolly in a "24 7 365" way ;) So even if there were memory leaks (and I am not even claiming that there might not be such leaks) they would only (presumably, and as far as I can tell so far) very slowly "eat up your available RAM". And then Windows would start swapping, so essentially if your system would start becoming "low on memory" everything else before that moment would have come to a "grinding halt" due to the intense memory swapping to/from your SSD (or even worse: a spinning disk). Or in other words: modern operating systems do everything to prevent such an "out of memory" exception condition.

And now we come to my working conclusion.... CHAPTER III...

till213 commented 11 months ago

CHAPTER III - MY CURRENT (WORKING) CONCLUSION

So as I wrote that I declared almost every method of mine to be "noexcept" ("I, the programmer, herewith certify that none of my methods will ever throw any exception, I swear by the cooler of my CPU!") - what happens if it (or any called function below) throws an exception after all?

Exactly!

According to the C++ standard the operating system is then free to "kill" the process in the most brutal way it sees fit: by simply immediatelly removing it from its memory, no questions asked!

And that is most likely what you are seeing, and that I can reproduce by - deliberately - throwing a "test" exception from anywhere within my code (that is, again, declared "noexcept" - "no exception thrown").

(By the way what is the purpose then to declare a method "noexcept"? Because if the compiler knows that a method is not throwing any exceptions it can then better optimise the compiled code: so the "noexcept" declaration is merely a strong hint to the compiler to generate better code for us).

So what I will do in the upcoming "test build":

I will let you know once this test build is available, as previously mentioned. Your help in testing it would be greatly appreciated!

till213 commented 11 months ago

P.S. I was fully aware of the problems with "noexcept", from the very beginning: in fact, it is on my to-do list to - gradually - introduce exception handling in my own code, especially in the import/export plugins where the current "error code handling" is indeed very tedious.

But this requires me to carefully check my code again whether it is really memory-leak free, especially when starting to throw exceptions (by consequently using RAII respectively ensuring that Qt is properly made aware of all "memory children" etc.). And all this is a bit lower on my to-do list, as I still want to implement / improve on existing / new functionality first.

Bloodrule commented 11 months ago

That is by far the best response I've ever seen to a msg I've sent for help! Thanks for such a comprehensive reply.

My system does indeed have MSVCRT.DLL, located in C:\windows\system32\

Also having got skyglobe.exe running by using Windows 8 compatibility mode, I noted that it did create a Sky Globe folder at C:\Users\Me\Documents and files are being stored in it.

I'll be happy to test drive the next version and to report back here. I can't pretend to have fully understood your exposition on memory leaks, having only rudimentary C language skills, but it's a great reference for other programmers.

Thanks for the pointer to Dependency Walker too - still coming to grips with it.

till213 commented 11 months ago

I will still provide a Sky Dolly version that „catches all exceptions“ (in the top-level main() function) and that will hopefully help in tracking down the root cause of the start failure. -> re-opened

till213 commented 11 months ago

"Also having got skyglobe.exe running by using Windows 8 compatibility mode, I noted that it did create a Sky Globe folder at C:\Users\Me\Documents and files are being stored in it."

Yes, your note about the Windows 8 compatibility mode is very valuable, thanks!

Just for the record: Sky Dolly is expected to run "out of the box", without any compatibility mode, including on Windows 11 (unfortunately myself I am stuck on Window 10 for the time being - Mac user here ;))

But what this tells us: I suspect that the premature "crash" is indeed related to some Windows 11 "security settings", perhaps to some "user access controls". Windows executables also know the concept of "manifests" that declare what an application intends to do, although I am not very familiar with those just yet.

Indeed, Sky Dolly essentially requests the following system resources upon launch:

Here is the thing: even if Sky Dolly was denied access to any such "resource" I am accessing all those resources via the underlying Qt toolkit (that not only handles user interface elements, but any other "operating system resources" such as files, sockets, databases etc. in a "cross-platform manner"). The point being: Qt is supposed to be an "exception free API", that is none of its functions is supposed to "throw an exception" (but instead catch them all internally and convert them to "error codes", as outlined above).

But before I speculate any further I am still extending the Sky Dolly code to catch any exception. That is not the problem: the problem is to get to know the "nature" of the exception. Believe it or not: any other programming language that supports "exceptions" allows you to query the location in the code (function/method name, line number etc. - essentially the so-called "stack trace" that tells you "who called who") - but not so in C++. Because C++ is "tuned for performance" and "you pay only for what you use" - and bla bla bla ;) (That's why we all love and hate C++ at the same time ;)).

Anyway, I don't want to bore you to death with those gory implementation details ;) So just to let you know that "I am on it!" :)

till213 commented 10 months ago

Hello,

Sorry for the long wait: dealing with "stack traces" (using the 3rd-party "Cpptrace" (https://github.com/jeremy-rifkin/cpptrace) took longer as expected.

However a new v0.16.3 release has just been made available:

https://github.com/till213/SkyDolly/releases/tag/v0.16.3

I will shortly also publish it on flightsim.to (same ZIP archive, as usual).

This release

I do have some idea what might go wrong on Windows 11 (and possibly also Windows 10) when "user access control" disallows an application to access the "Documents" folder without prior permission by the user. Therefore the app needs to have a "Wants to access the Documents folder"-claim in its "manifest" file - Sky Dolly does not have such a claim (rather the tooling / Qt that I use presumably generates some blank "default manifest" file).

Anyway, in order to verify my assumption you could be of great help if you tested this release! If my expecations are correct then you should now get a "crash handler" dialog, with hopefully enough information (the expected "exception", for instance) that helps me track down the root cause!

Here is a dialog with a (deliberate) "example exception" of mine:

Example-Exception

So ensure that you can reproduce the "Sky Dolly not starting" situation with the previous v0.16.2, specifically:

Once you are sure that Sky Dolly v0.16.2 does not launch anymore (and also do not launch it "as administrator" - Sky Dolly should never be run "with administrator rights"), delete it (or move it to some other safe place) and unzip the latest Sky Dolly v0.16.3

I expect it to also not launch, but now you should (hopefully) get a "crash handler dialog", as shown above.

Then:

The "report" also contains the path to your documents folder - including your user name. Feel free to edit that part (like c:\Users\XXX\Documents\Sky Dolly) if you do not want to reveal your real name. The most important info would be the stack trace and the exception information that we (hopefullly) get.

If you could provide this report that would be so helpful, so thank you in advance!

Bloodrule commented 10 months ago

I followed your instructions to the letter including rebooting before trying the new version. Version 0.16.3 also failed to do anything (no error message) but opened normally when I ran Windows 11 compatibility troubleshooter which had recommended trying Windows 8. Opting for Windows 8 meant that it opened normally. But here's the weird thing - I went back and disabled the Windows 8 compatibility option and next time it opened normally - having not opened on multiple attempts prior to trying Windows 8 compatibility. And since then it's continued to open normally. At no stage did I see the error message that you reproduced in your latest message. Not sure if this helps or not!

till213 commented 10 months ago

Thank you for your quick feedback, very much appreciated!

"Version 0.16.3 also failed to do anything (no error message)" - hmm, bummer. That kind of diminishes my "user access control - no access to folder Documents granted"-theory. I would really have expected some kind of exception in this case. Well, actually not really, as the underlying Qt framework should actually "shield" me from such OS-level exceptions (I am opening all files via the corresponding Qt file API that is never supposed to throw any exceptions).

However this theory is still not completely off the table, because when you enabled "Windows 8 compatibility" mode I guess Windows then "remembers" for this specific *.exe (respectively executable path) that you gave permission to the Documents folder (implicitly by switching to Windows 8 compatibility mode (?)). So when you turn off "Windows 8 compatibility" then your executable still gets access to the Documents folder.

In any case, "ChatGPT" suggest to add the following section to the manifest XML file that is embedded into the executable (and that currently gets "auto-generated" - somehow - in my case, I strongly assume):

<?xml version="1.0" encoding="UTF-8"?>
   ...
  <application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings>
      <requestedExecutionLevel level="asInvoker" uiAccess="false" />
      <requestedPrivileges>
        <requestedPrivilege name="documentsLibrary" />
      </requestedPrivileges>
    </windowsSettings>
  </application>
</assembly>

Or in other words: "request the privilege" to access the documents "library" (folder). What should happen then is that you see a dialog (from Windows 11 itself), asking you whether it is okay that Sky Dolly accesses your documents. Of course Sky Dolly should gracefully continue if you decline this dialog.

Btw. I am mostly developing Sky Dolly on macOS ("just because" - no particular reasons other than this is my main operating system in my private time): macOS knows this "can app XYZ access your documents?"-concept since a long time:

grafik

Now the price-winning question is of course "what happens if the user declines the access?". Well, Sky Dolly properly checks whether the folder could be opened successfully or not, and if not:

grafik

It then launches properly, but all "recording" etc. functionality will remain disabled, until the user explicitly opens / creates a new logbook, via the File | New... and then the file chooser (note that the file chooser will implicitly grant Sky Dolly access to any file the user chooses, as this is a deliberate choice by the user to do so).

Or in other words: Sky Dolly is already prepared for this situation, and properly checks any file access whether it was successful or not. This further "diminishes" my theory that the problem is related to "user access control" (UAC) on Windows 11.

So I will do what every developer does if there is a problem without apparent solution: I will first go on holidays and rethink the whole problem :)

In any case I will add the above "request privilege" to the application manifest and think about any other possible reasons. Once the manifest has been extended accordingly I will then close this issue - unless I have some more ideas after my holidays.

Because even ChatGPT doesn't give me any concrete hints what could go wrong when "app fails to launch on Windows 11 without error message":

https://zzzcode.ai/answer-question?id=54e78445-4daf-47e4-8437-359aca085b68

Also, the next v0.17 release will ship with a newer Qt 6.5.x library, so perhaps this will also iron out some "quirks", who knows...

PhilM5253 commented 10 months ago

Hi, just a quick message to say that following my request for help and our conversation on flightsim.to comments, I have joined this GitHub conversation, hopefully to find an answer to my 'frozen' copy of Sky Dolly. Still no luck making or opening a logbook, and hence still no recording.

I have looked at the regedit point you mentioned above: my system does have SD / till213 entries. Would it help for you to see a screenshot of those ...

Screenshot skydolly regedit
till213 commented 8 months ago

hopefully to find an answer to my 'frozen' copy of Sky Dolly.

Hello @PhilM5253 , sorry I just noticed your comment here now. What exactly do you mean by "frozen Sky Dolly"? As you have actual registry entries, as created by Sky Dolly, this means that Sky Dolly actually launched (at least once).

This issue however is about a different problem: Sky Dolly did not launch at all for the original reporter. Setting the Windows 8 compatibility mode (right-click on SkyDolly.exe) did allow Sky Dolly to launch. After that unsetting the Windows 8 compatibility mode kept Sky Dolly launching.

I did not figure out what exactly is responsible for this behaviour, as no exception was ever thrown during startup (which was my initial guess: so I added an "exception handler", but that was seemingly never called, presumably because Sky Dolly was never ever launched at all).

However I do not know what you mean by "frozen": can you please create a separate issue, and describe in detail what exactly is "freezing"?

till213 commented 8 months ago

I am actually now closing this issue, with the following summary: