Nexus-Mods / Vortex

Vortex Development
GNU General Public License v3.0
901 stars 131 forks source link

Suggestion - Please make a Linux native version #2862

Closed VortexFeedback closed 1 year ago

VortexFeedback commented 5 years ago

Vortex Version: 0.16.15 Memory: 15.61 GB System: win32 x64 (6.1.7601) Please make a Linux native version so that I don't have to just run it through WINE.

Reported by: AmayaSasaki

8BitCerberus commented 5 years ago

Especially with Valve's Proton/Steam Play now getting so many games, including many serviced by Nexus' mods, running as or nearly as easy as on Windows. And for non-Steam games there's Lutris that takes a lot of the hassle out of managing Windows games on Linux.

TanninOne commented 5 years ago

Vortex is built on technologies that are also available on Linux (and MacOS), except for a few native libraries that would have to be re-implemented (stuff like reading the application version from a .exe file). It's mostly a matter of investing the time, something we can't do at the moment.

TheMatthew commented 5 years ago

I was looking through the code "lightly" I don't pretend to know what I'm doing/saying.

It appears as though the only "windows" dependency left is Nexus/wholocks which could be replaced by https://www.npmjs.com/package/@ronomon/opened or something similar.

Everything else at the moment looks OK.

I am not sure when I will be competent enough to make a PR. if there are any Node/typescript masters out there, this could be a cool feature.

VortexFeedback commented 5 years ago

Vortex Version: 0.16.15 Memory: 15.66 GB System: win32 x64 (6.1.7601) I see Vortex is an electron app, with the improvements being made to Proton/Steam Play it would be incredible if there were a native Linux version of Vortex to use with Steam Play games.

Or at the very least some documentation put into getting every thing up and running under Linux, since Vortex does seem to run fairly well under WINE.

Reported by: Dreae

TheMatthew commented 5 years ago

I doubt that steam play / wine would work well on vortex as it uses undocumented windows apis. Just a heads up.

8BitCerberus commented 5 years ago

It works very well, actually. You can either install it directly in your game's prefix to manage that game (and subsequently install it in to each game's individual prefix that you want to manage), or set up it's own prefix and use symlinks to point it at each game install. I've currently got it managing Skyrim SE and installing mods is no problem. The only real issue is SKSE64 (and F4SE) doesn't work under Proton or Wine yet. There is a patch you can use and manually compile both Wine and Proton to get the script extenders working. But Vortex itself works pretty much flawlessly.

Since Vortex is using sym/hardlinks instead of MOs virtual file system, a native Linux version should just need to know where the games are installed and use Linux's link command instead of Windows' (ln vs. mklink).

TheMatthew commented 5 years ago

There's a magnificent individual that did this https://github.com/Nexus-Mods/node-wholocks/pull/2 I got vortex working in wine, so it "works" but a native version would still be an interesting prospect for some.

Mumrik93 commented 5 years ago

Getting Vortex to work on Linux would really kick up Linux as a viable alternative for gamers who are tired of Windows.. of which there are many, myself included.

codeman101 commented 5 years ago

@TanninOne

I'd love to assist this issue and devote time to solving it but a hurtle I'd have to overcome that I'm not willing to do at the moment is going through the process of learning TypeScript. I'm curious as to why that was decided to be the base language of the project vs C/C++, C#, Java, Python ect?

TanninOne commented 5 years ago

TypeScript is a much easier language to learn than C++, C# or Java and less "unusual" syntax wise than python. The decision was made a) to encourage contributions from less experienced developers because if you already know one of the above mentioned languages you can start writing useful typescript code within hours b) because a customisable UI was one of the top requirements for the project and HTML/CSS (vortex is based on electron) is more readily customisable and better known to more people than something like QML or XAML. c) If this project had been written in C# (which was in fact an option we discussed), it would be using WPF and XAML and a Linux port would be entirely out of the question. If it had been written in C++ (another option) there would be a lot more code that would have to be ported for Linux (or MacOS). Java is never a good choice for UI-heavy, close-to-the-system applications (A lot of what Vortex does is managing files, creating symlinks, fixing permissions and such, I wouldn't want to do all that through JVM...). And python - what would be the advantage of python over javascript?

codeman101 commented 5 years ago

A few things.

  1. As far as python over javascript I'd think the syntax is easier for a beginner. (but I agree aside from Qt as of recently, there's no simple framework to write a GUI with python)
  2. If the reason the project wasn't made in C# was because it'd be very hard to bring over to Linux why wasn't it designed with the intention of bringing over to Linux in the first place?
  3. If the project is based off electron why is it using typescript instead of javascript? I thought all electron based apps used javascript for their UI and by their nature were cross-platform (for example slack) I even watched a youtube video a while back making it sound like the big plus to making an electron based app was that it would automatically be cross-platform. So you can imagine my confusion.

Update As far my third point I started going through the list on electron wiki of electron based apps and realized their not all javascript based. I'll add it to my to-do-list to try to help with making this project Linux native. :)

Patricol commented 5 years ago

@TanninOne

I'd also be interested in working on this; but unless I can figure out which components are os-specific without needing to read through the whole project; it'll be hard to prioritize.

If you can point out any of the relevant areas in the code; it'd greatly lower the barrier to contributing.

Or potentially add inline markers (e.g. //TODO: fix for linux) noting os-specific things as you run into them during the work you're already doing on the project. Or any other thing you'd prefer to do that'd achieve a similar result.

codeman101 commented 5 years ago

@TanninOne

In addition to what @Patricol said can you confirm whether or not having a Windows system is a requirement to figure out some stuff? (I don't run one anymore other than wine)

Patricol commented 5 years ago

@codeman101 We'd need to test that our changes don't break the Windows version, so I'd say yes.

TanninOne commented 5 years ago

A few examples of how Vortex might require and use platform dependent code:

Tbh.: My advice to anyone trying to work on a Linux version: Start by setting up a fork for working on Linux compatibility, start by trying to build it. Investigate build errors if there are any, fix them in a way that hopefully doesn't affect the windows build, when you have a running build, test the stuff that is intuitively more low-level (stuff like getting at the icons for executables, searching disk, creating links) Don't start looking at the code trying to figure out what needs to be done. Once you have it working natively on Linux, see if you can reduce the difference to the official build and then we can check together if anything might affect Windows compatibility before merging it into the official fork.

Also: the parts that are most likely require work are native modules we created, not the Vortex repo itself. look at the dependencies in package.json, our custom modules are the ones that aren't listed with a version number but with an identifier for a github repo. Stuff like

are bound to require platform-specific code that is either not implemented for Linux or not tested.

codeman101 commented 5 years ago

@TanninOne

Tbh.: My advice to anyone trying to work on a Linux version: Start by setting up a fork for working on Linux compatibility, start by trying to build it. Investigate build errors if there are any, fix them in a way that hopefully doesn't affect the windows build, when you have a running build, test the stuff that is intuitively more low-level (stuff like getting at the icons for executables, searching disk, creating links) Don't start looking at the code trying to figure out what needs to be done. Once you have it working natively on Linux, see if you can reduce the difference to the official build and then we can check together if anything might affect Windows compatibility before merging it into the official fork.

I made a fork of the project last night. Although at the time it was out of anger over the comment Patricol made in response to my question about needing to run windows. Although now thinking about it more I suppose I can run a VM of a windows preview build to test it.

Due to the fact that the program is using the registry to detect the games the only way I can see to make it cross-platform is to have the program detect which OS it's running and do a recursive search of the users home directory for the Linux implementation.

Edit: Also you claim that having the program use the registry to find games is faster but I just remembered from when I ran windows that although that may be true it didn't get all of the games.

I realize now that the design trade-off is "should we do an implementation that would make game searching really fast and use something that is windows dependent or do an implementation that would be slower but more cross-platform friendly." Personally I don't feel the the correct tade-off decision was made for two reasons.

  1. The current implementation of the registry method doesn't get all of the games
  2. I never considered the game search method to be much of a plus over vortex's predecessor. The main plus I realized with vortex over it's predecessor is the ease in the way mods are added. The slowness in the search didn't bother me nearly as much compared to how mods were added.
TanninOne commented 5 years ago

@codeman101 we're not debating design decisions here. Just let me make one thing very clear here: I'm open to a Linux version but no trade-offs will be accepted for the Windows version. 0. Zilch.

second: Please educate yourself. Seriously, it's getting pretty annoying discussing with you because you insist on discussing about things you don't fully understand.

Vortex has custom detection code for all games, usually with fallbacks. Every game that Vortex tries to detect through the registry does get detected immediately (within less than a millisecond), the only people having problems with that are those that insist on mucking about manually in their registry or move games around manually instead of doing it properly. So 99% of our users get all their games discovered within milliseconds, a full disk search like NMM did (which is also an option in Vortex!) can take hours on a large disk. The claim that this is worse than just having a disk search is either misinformed or insane. The fact you don't even seem to be aware that Vortex already supports searching the disk for games also shows you have spent very little time with the tool before coming here to tell us what to do.

You seem intend to construct an excuse for changing core functionality because you think it would make cross-platform support easier. a) It wouldn't b) it will not happen anyway. Again: Windows is the primary platform, absolutely no concessions will be made to the Windows support just to make Linux support easier. Either accept that and try to be constructive or please go away.

Patricol commented 5 years ago

Although at the time it was out of anger over the comment Patricol made in response to my question...

@codeman101 My apologies; I didn't intend for my last comment to be abrasive in any way.

codeman101 commented 5 years ago

@Patricol

All good. I feel like some good has come out of me making my own fork because it's forcing me to learn the Windows API which has always been on my to do list anyway for the desire to help wine development.

@TanninOne

I agree. I will not argue with you anymore. Let's just agree that we have fundamental differences as to how we would've approached this project and move on. If I'm able to convert the necessary submodules to Linux code (which by the way aren't the ones you listed above) I'll post links to my changed files here. Unfortunately that's the best I'll be able to do as I copied a branch that makes the project ad-free. (one of the fundamental differences) Again I don't want to argue anymore I'm just explaining why my changes won't be mergable. Since this difference prevents my changes from being mergable anyway I'm going to try to covert the code into a pure Linux version (overwriting the Windows API code)

All and all I'm glad I forked this because as I said before it forces me to learn the windows API so even if someone beats me to punch of making this Linux compatible at least I got quite a learning experience out of it.

Oh also I'm sorry for what I said in the pull request. I didn't read the whole post of your first response and I've come to agree with you about that request.

Patricol commented 5 years ago

@codeman101

Just try to be more understanding. We're all human, and attacking people (and/or their views) left and right will get you nowhere in life; regardless of whether or not you are correct.

codeman101 commented 5 years ago

@TanninOne

While trying to work on making the project Linux native I realized something and made this issue on proton (explained in issue) https://github.com/ValveSoftware/Proton/issues/2670 and because of that I've decided to delete the fork I made since as far as I can tell it would only benefit native Linux games.

To future readers who still want to make this native for Linux

Search

node-

in the Nexus-mod github page to find all the modules that have this header file.

TanninOne commented 5 years ago

esptk, bsatk and ba2tk are all for gamebryo games (TES, Fallout) which don't run natively on Linux anyway.

Tbh if you're playing a game through wine on Linux, I would also run Vortex through wine and use native Vortex only for native games - things are just going to be too complicated otherwise. For example: Vortex uses LOOT to order plugins for the gamebryo games. LOOT being a tool exclusive to windows games itself works under the assumption it's on windows - having stuff like case-insensitive file name comparisons hard-coded into it - same with the "gamebryo-plugin-management" vortex extension that uses it. Another example: Vortex has to manipulate ini files for certain features (e.g. to have separate save directories for different profiles in gamebryo games). Now ini is not a standardized format, implementations differ in how they handle comments, multi-line strings, encodings and so on. Vortex uses a library called "vortex-parse-ini" with a plug-in interface to support different ini-styles. The only one implemented right now is the one used by the windows api and we use winapi functions directly to ensure perfect compatibility with games that also use those apis - like the gamebryo games. On Linux you won't have those apis available, so you have to use some other ini format. Easy enough to implement but it may not be 100% compatible with the game, which could then lead to obscure bugs where ini settings don't stick or stuff like that.

I imagine there is a lot of grief down that rabbit hole.

codeman101 commented 5 years ago

@TanninOne

esptk, bsatk and ba2tk are all for gamebryo games (TES, Fallout) which don't run natively on Linux anyway.

Oh ok.Well if someone tries to make a native Linux version of vortex they'll need to do something about them because yarn build fails because of them being there.

Tbh if you're playing a game through wine on Linux, I would also run Vortex through wine and use native Vortex only for native games - things are just going to be too complicated otherwise.

Yes I know but I didn't realize it until my previous post because I was actually trying to run TES via proton from the native steam client and then realized it'd make more sense to install vortex into that proton install of TES (hence the issue I linked to in previous post)

Another example: Vortex has to manipulate ini files for certain features (e.g. to have separate save directories for different profiles in gamebryo games). Now ini is not a standardized format, implementations differ in how they handle comments, multi-line strings, encodings and so on. Vortex uses a library called "vortex-parse-ini" with a plug-in interface to support different ini-styles. The only one implemented right now is the one used by the windows api and we use winapi functions directly to ensure perfect compatibility with games that also use those apis - like the gamebryo games. On Linux you won't have those apis available, so you have to use some other ini format. Easy enough to implement but it may not be 100% compatible with the game, which could then lead to obscure bugs where ini settings don't stick or stuff like that.

Fortunately since the games aren't Linux native anyway this isn't an issue since you'd have to use a wine backbone to run the game so you can install the windows version of vortex into that bottle. (which is a feature being worked on for proton see link above)

Side note: I'm going to see if the wine database has reported this bug but I was reminded by my computer tonight that currently vortex doesn't run on wine past installation regardless of install the dotnet version mentioned in the one review on the database.

Edit: I made the bug report for it on the wine database.

TanninOne commented 5 years ago

esptk, bsatk, ba2tk and gamebryo-savegame are all libraries pulled in from extensions: gamebryo-plugin-management, gamebryo-savegame-management, gamebryo-archive-invalidation, gamebryo-... I would disable those extensions entirely, that way it will not try to build the extensions either.

The best place to do that is probably in "BuildSubprojects.json" and "BuildSubprojects.js". The js is the code, json is the data file which specifies which subprojects (mostly extensions) to build. I would add a new "platform" parameter there, something like platform: ['win32', 'linux'] Of course this would have to be implemented in the js file: "if build platform not in list of platforms, skip subproject"

Then, to start off, just mark all subprojects as windows only, try to build. Get the core application to build and run on Linux, then check which extensions even make sense for native Linux and enable/fix them one-by-one.

Freso commented 5 years ago

LOOT being a tool exclusive to windows games itself works under the assumption it's on windows

As one of "the LOOT people", I’d just like to point out that LOOT actually has built Linux native binaries for years (available on Bintray). While it is true that it by default assumes Windows‐ness, it shouldn’t err out if not run on Linux and should be able to sort load orders for Wine/Proton installed games given the right settings. I’m not sure if we’ve actually had anyone do this yet, but it should theoretically be possible AFAIK. :)

(This also means it should be possible to also use LOOT(/loot-api) on a Linux‐native Vortex build, either for modding Proton-CK games or for something like OpenMW. If there are any issues with that on the LOOT side of things, please let us know!)

ScumbagDog commented 5 years ago

The best place to do that is probably in "BuildSubprojects.json" and "BuildSubprojects.js". The js is the code, json is the data file which specifies which subprojects (mostly extensions) to build. I would add a new "platform" parameter there, something like platform: ['win32', 'linux'] Of course this would have to be implemented in the js file: "if build platform not in list of platforms, skip subproject"

Then, to start off, just mark all subprojects as windows only, try to build. Get the core application to build and run on Linux, then check which extensions even make sense for native Linux and enable/fix them one-by-one.

I just tried doing exactly this, I actually got the main UI working. That being said, the dashboard threw this error when it tried to render, and I can't actually set a game mode (as well as a slew of minor issues).

TanninOne commented 5 years ago

Oh right, it should be safe to put the "setUserTasks" into a if (process.platform === 'win32') check, it's nothing important.

@Freso thanks for the info, I was actually not aware of the LOOT linux builds, sorry for the misinformation in my earlier post then.

ScumbagDog commented 5 years ago

I've stumbled upon another "interesting" issue. Running yarn run build doesn't make a proper binary in the out folder, and through debugging the dist command (which does produce a binary, but still throws errors in doing so, full error dump here.) I received the following error:

error TS5042: Option 'project' cannot be mixed with source files on a command line.

I'm not particularly well versed in JS (or TS) development, so I am not quite sure how to fix it, but I'm fairly certain there's an issue with the configuration of the app-builder-bin module.

Edit: It actually appears to just be tsc that doesn't like me putting the DEBUG= flag after it, so disregard that error message

ScumbagDog commented 5 years ago

This is a pretty decent start, though it isn't particularly usable yet. Do you happen to know what .net is used for, and if it even makes sense to try and use mono as a replacement?

TanninOne commented 5 years ago

.net is used for most mod installations, that is: determining which files to extract from the mod archives and where to put them, especially when installing mods with scripts (fomods).

There are two kinds of fomods: those based on xml files and those based on csharp scripts.

The xml based ones should work with mono. Not sure if the code needs any patches to Nexus-Mods/fomod-installer but it should be very doable. Of course one could also port that code to JS or something but that'd be a lot of work.

The csharp scripts however run essentially arbitrary code to create a ui, copy files and so on. These may be harder to get working with mono but more importantly they may be a security risk. On windows (Vortex, NMM, MO) the script is run within a sandboxed AppDomain which limits file permissions the script has and so on to limit the damage a malicious or buggy script can do. To my knowledge this sandboxing is not supported in mono and .net core (MS has stated that they consider it a mistake and insecure so it's unlikely they will add support to core in the future)

So my suggestion would be: Simply refuse to support c# scripted installers on linux. That's what I'd do. Or at the very least tell the user about the potentially security problem. Those installers are mostly used with fallout 3/nv and have come out of fashion with later games fortunately.

ScumbagDog commented 5 years ago

So my suggestion would be: Simply refuse to support c# scripted installers on linux.

I think this is a good start. Or rather, a good milestone after fixing the BSA-extensions (not completely sure how many of them work, but they use a windows-header in their code, which may or may not be horrible to patch out). As for the C# installers, mono does appear to have a sandbox, but I don't know how well it actually works, so focusing on XML-installers is definitely more lucrative to me.

ScumbagDog commented 5 years ago

On a sidenote: I get some errors when running the test suite (primarily in src/util//Steam.ts where app is undefined). Should I just ignore these for now and work on the broken extensions, or are they actually important?

TanninOne commented 5 years ago

That looks like it could be what we need re mono sandbox, yes. Maybe I had outdated information about mono. But yes, xml installers are far more common. The only big mod I can think of for Skyrim that has a c# installer is skyui.

Regarding the unit tests: It depends on the issue but yes, I'd ignore it for now. The problem with Steam.ts I think is that on windows it retrieves the steam config location from the registry, on other platforms it uses an electron function to determine the home directory and derives the path from that. Unit tests aren't run in electron but in native node.js though, so that function isn't available. However, we don't have unit tests to test the Steam functionality itself anyway, it's just imported as a "collateral". Solution will probably be to create mocks for those functions because we don't want unit tests to access real system information anyway.

elken commented 5 years ago

Might be possible to use this library to execute C# scripts and wrap the scripts in mono sandbox.

As for unit tests I agree using a SteamProvider to mock would be the best solution.

MaverickMartyn commented 4 years ago

Hey everyone.

I just read through all of this, and it seems to me that some good progress has been made on the port. I am not sure how much help I could be, but if anyone could use it, I gladly offer my assistance here.

TanninOne commented 4 years ago

I've recently made a bunch of fixes re linux support and tested the build so while we still can't offer official support and some features will probably not work correctly, Vortex will correctly build and run on Linux natively atm and at least the installation of mods is tested to work.

breversa commented 4 years ago

I'm willing to try a test release when there's one. :-)

caysilou commented 4 years ago

Good to hear! I will be very happy to test this.

DMNerd commented 4 years ago

Atleast AUR build then?

Oscar-Macdonald commented 4 years ago

Sounds exciting! Tempering my expectations, but still thrilled to hear that this is a thing.

Weirdo1312 commented 4 years ago

I support this <3

ScumbagDog commented 4 years ago

If any of you fellow tuxmen are having some fun™ installing fomod-installer as a dependency, it might be because the msbuild library used in the module uses xbuild on linux and MacOS, which is a now-deprecated build tool for Mono. Installing your local mono-devel package should fix it, at least for now.

kattjevfel commented 4 years ago

Looking forward to give the linux version a spin :3

monstoor commented 4 years ago

As an experiment I have compiled this on my OpenSuSE Tumbleweed system. The main errors I am getting are: 1) "Failed to bake settings files: winapi.GetPrivateProfileSectionNames is not a function" when I 'activate' any installed game. 2) "Failed to suggest path: winapi.GetVolumePathName is not a function" when trying to set a path for the mods. 3) I cannot activate the Gamebryo Plugin Indexlock. I have not coded for many many years, so any suggestions would be appreciated!

TanninOne commented 4 years ago

re 2: good catch re 3: None of the gamebryo games run natively on linux so that extension doesn't have to be supported anyway re 1: Same, the code you're getting this error from shouldn't get run on Linux in the first place because it relates to games that don't exist on Linux.

Freso commented 4 years ago

FWIW, most of the Gamebryo games run or can be made to run near-natively via Proton. Ie., you click "Play" in Steam and they run and you never notice that they’re not actually Linux native. It would be great if these could be supported by Vortex too.

TanninOne commented 4 years ago

No, trust me, you don't want that. If you run the game through an emulator (and yes, wine is an emulator despite the name, it's just emulating an api instead of hardware) you want to run Vortex through an emulator as well.

The ini functionality is a perfect example (not the only one!) for why: The ini format is not standardized, if you use a library, that library may generate different files from what the windows api can consume. The games use windows api to read/write their settings. Vortex uses windows api to read/write the ini files for those games. As a result the format is guaranteed to be compatible. Wine does some magic to emulate those functions on linux, how it works is a black box to us, all we rely upon is that the api is compatible. Now if native linux Vortex were to use native functionality to generate ini files that are supposed to be compatible with the game running through wine, effectively we would have to write code that tries to emulate how wine emulates the windows api. That's madness, you put an emulation layer into Vortex just so you don't have to use the existing emulation layer (wine). What this does is cause us a ton of work just so linux users can enjoy fancy new ways for things to go wrong.

Ini files are just one example, what if the game stores relevant information in the registry? We can't access the registry through native Linux code. And can we rely on symbolic links created in Linux natively to work correctly within wine?

And then there are external tools like FNIS that Vortex utilizes that are only written for Windows because the game only exists on windows. Even when Vortex is fully functional on Linux, the environment surrounding modding for those games is not.

So my advice, and the only approach we could ever consider supporting officially is: use emulated vortex with emulated games, use native vortex only with native games.

kattjevfel commented 4 years ago

Running Vortex through wine is a absolute pain though, but pick your poison I guess.

MattSturgeon commented 4 years ago

Running Vortex through wine is a absolute pain though

It's actually pretty simple with something like this lutris installer.

TanninOne commented 4 years ago

If running Vortex through wine is a problem then that is a separate thing to investigate but that doesn't invalidate my argument.