search

sonic2kk / steamtinkerlaunch

Linux wrapper tool for use with the Steam client for custom launch options and 3rd party programs
GNU General Public License v3.0
2.17k stars 73 forks source link

⚠ MO2 - Mod Organizer 2 overhaul discussion #598

Closed frostworx closed 1 year ago

frostworx commented 2 years ago

This issue is meant to be a generic brainstorming issue for a possible MO2 overhaul. Feel free to contribute any ideas you have to improve it. Of course PRs are very appreciated as well as usual.

Starting with this proposal: If dotnet is no longer required at all for MO2 (can someone confirm it is no longer required for modding?), it might make sense to simplify the whole implementation by using one global "install directory" (=MO2 Program Files dir) and f.e. symlink it into the corresponding compatdata without any further installation. Imho this would help massively to solve almost all problems people have with the confusing "gamemode" and "standalone modes".

har-nick commented 2 years ago

Thank you detailing the install process out for me! Set MO2 up the same I did previously and downloading through the Nexus works as expected!

sonic2kk commented 2 years ago

Thank you detailing the install process out for me! Set MO2 up the same I did previously and downloading through the Nexus works as expected!

That's amazing news!!! I am hoping to get this branch merged in sometime in the next couple of weeks so that is awesome to hear! Hopefully we can get more tests, and on top of that I'm going to test with my existing mod installs when I get the chance. If all works as expected I'm planning to have this merged before the end of the month and in place for STL v12.

Please feel free to report any issues you encounter as the branch is still in development :-)

sonic2kk commented 1 year ago

I haven't gotten about this :sweat_smile:

This branch is almost 2 months old, at this point I don't think there will be many more testers, and the branch has seemed to have some positive improvements.

The plan is to resolve the merge conflicts (of which there are quite a few iirc) and then do some quick testing to ensure the following:

  1. New, fresh mod setups work
  2. Existing mod setups work
  3. There are no other regressions from merging

I will probably spend the next 3-4 days seeing that the Hedge Mod Manager support goes over smoothly enough, and fix any other reported issues as best I can. Then I'm planning on taking a bit of a break, probably 2 weeks or more, to enjoy some games and chill out for a while before merging in the mo2-overhaul branch. I don't want to merge it in and then disappear.

This branch will make its way into v12 if there are no significant problems :+1:

sonic2kk commented 1 year ago

On break, but a draft PR #674 is up now. Testing is still needed on my end, but the conflicts are resolved.

Probably gonna get it merged in a few weeks from now.

tetoNidan commented 1 year ago

Not sure if this is the right place to suggest this but since windows is not case sensitive and linux is, I have been creating links for folders in multiple cases. Mod creators dont really have to stick with any naming convention. Dont remember which mod might have been for oldrim that used Skse as their folder name for SKSE. Things can get out of hand in the data folder when you have to manually install some mods.

$ ln -s SKSE skse

$ ln -s Scripts scripts

$ ln -s Plugins plugins

$ ls -s Meshes meshes

$ ln -s Tools tools`

Dont know it this helps at all. Not that big of a deal unless you manually install something though. Like some FNIS mods or maybe an enb or something.

Also I was thinking about setting up a new play through of FO4. If I can help with testing your new MO2 branch without messing with my current play through of Skyrim AE I would gladly do so! Where can I find instructions on how to use your testing branch?

Side note I noticed Skyrim AE was crashing after I reset the prefix with STL and it was because I had the Skyrim SE upgrade version. Fixed it by setting steam comparability to a proton version of gloriouseggroll then launching and letting Skyrim upgrade to anniversary edition. Not much you can do to fix that but might be helpful to know if someone complains about crashing after new pfx with AE upgrade version.

Thanks for all you all do. Love me some STL.

sonic2kk commented 1 year ago

Hehe, glad you like STL! Nice to see some positive comments :smile:

Yes, the folder naming is certainly an issue and it's something I've kept in the back of my mind. Perhaps there is a way to force all folders recursively in a given mods directory to be lowercase, but I'm not sure if 1) this could cause problems with some mods, and 2) if there is a reasonable way to do this. However, it's 100% a valid thing to bring up, and at the very least it is something I will keep in mind to add to the wiki. It isn't an ideal manual step, but documenting it is better than not, right? :-)

Also I was thinking about setting up a new play through of FO4. If I can help with testing your new MO2 branch without messing with my current play through of Skyrim AE I would gladly do so! Where can I find instructions on how to use your testing branch?

Awesome, thank you! Testing the branch can be done in a few ways:

A heads up too: You may need jq for this branch to work -- various parts of STL that call out to web services have become reliant on jq to parse JSON, it may not be needed here but it's better to ensure you have it than not :-)

Thank you again for your willingness to test! Full disclosure, I have not had an opportunity to test this myself yet, but a couple of users have reported success (and no users that tested have complained yet more importantly :wink:) so I am hoping for the best here too!

While I'm hoping to get the branch merged soon, it is probably still about 2 weeks away before I will even be fully ready to push the merge button heh. So take all the time you need with testing.

Thanks!

har-nick commented 1 year ago

@tetoNidan I don't believe SKSE is picky about the case sensitivity of its folders, but from what I hear Proton may be, which results in duplicate folders when different mods use different cases like Skse/SKSE/skse.

If you like living life on the edge, you can recursively rename your skse folders with something like this:

find /PATH/TO/MO2/MODS/FOLDER -maxdepth 2 -type d -iname skse -execdir echo mv {} skse ';'

It'll find all folders named SKSE without case-sensitivity, and rename them using mv.

-execdir runs the command in the containing directory of the found directory. -exec would run it from the MO2 mods folder causing mv to fail. Also, SKSE folders are top-level, so -maxdepth 2 stops it delving into mod folders.

(Note that's a dry-run. Remove echo from that command when you're sure.)

tetoNidan commented 1 year ago

A heads up too: You may need jq for this branch to work -- various parts of STL that call out to web services have become reliant on jq to parse JSON, it may not be needed here but it's better to ensure you have it than not :-)

is JQ a jquery lib? id there something I can do for you all in python? Returning json i mean?

Also was thinking about your UI? Are you guys set on YAD? I could give a shot at building a GUI in wxPython. wxPython might be a little overkill but might be nice in case STL is ported to run on windows in the future. The reason I was thinking wxPython was the tabbed window interface for grouping relative settings apart from the whole? Might be nice but might not be something you want to undertake. Been looking for a new project to get me back in the coding game. The biggest problem I can see using wxPython would be compiling a single file executable to pull in all the WX libs every time STL updates. Might not be what you all want to be doing? Thought I'd ask though.

I will try my best to test the dev branch this weekend.

Thanks STL team!

har-nick commented 1 year ago

is JQ a jquery lib?

jq is a JSON processor. It's not especially big, but it's used I believe by Proton to format JSON responses when fetching updates.

This is mostly from memory and checking b731f7e4990dcfd6d96aabb4e4ea6ebe8f43a582, so excuse me if I'm mistaken.

Also was thinking about your UI? Are you guys set on YAD? I could give a shot at building a GUI in wxPython. wxPython might be a little overkill but might be nice in case STL is ported to run on windows in the future.

Not to diminish frostworx amazing efforts with this repo, but to sum it up, STL is basically a giant bash wrapper for various WINE functions.

Given that running Bash scripts on Windows would require WSL, and WINE and its functions aren't necessary on Windows, I'm not sure STL would benefit from a Windows release.

sonic2kk commented 1 year ago

Thanks @har-nick, your overview here even down to the specific commit hash is totally correct. jq despite its name is a JSON processor, and yes it's used for parsing Proton versions. It is somewhat unfortunate to mandate it as a dependency but to be totally honest, it's a small dependency and saves a bunch of headaches. It just makes things much more straightforward from a development and maintenance standpoint to use an actual JSON parser.

You're also entirely correct that SteamTinkerLaunch is a big Bash wrapper. This is why we are unfortunately stuck with Yad - SteamTinkerLaunch, to my understanding, started out a lot smaller than it is now and ballooned in scope. Now a full rewrite would be required given how much of the logic and Yad code are intertwined.

And once more, you're exactly correct on the Windows stance. SteamTinkerLaunch is just not needed on Windows. STL is meant as a wrapper to allow for configuration of Linux tools and Wine settings, so I also don't see the benefit of having SteamTinkerLaunch on Windows. If other users do see benefit and want to get STL working with WSL, by all means give it a shot! It can be good fun to just make something for the fun of it and I certainly would not want to take that away from anyone. SteamTinkerLaunch is open source under GPL-3.0, so long as you don't violate the terms of the license I also have no reason to stop anyone from doing whatever they want here - Open source rocks for a reason :wink:

So, at the risk of repeating much of what @har-nick has already pointed out, to give a response more directly to you @tetoNidan:

id there something I can do for you all in python?

I'm a Python guy at heart and there was some brief discussion about pulling in Python as a dependency to run some scripts (and also to run a custom headless version of #467 in Python, before plans changed on how to implement this), but in the end it was decided to keep SteamTinkerLaunch as light as possible. So while I appreciate the offer here, it's all good :-)

Are you guys set on YAD? I could give a shot at building a GUI in wxPython

Like with everything you said here, I can't stress enough how much I truly appreciate the effort and initiative here. But as I and @har-nick went over a little above, we are sort of stuck with Yad. STL would need to be entirely rewritten, and honestly having worked with it for about 9 months in-depth, a lot of what SteamTinkerLaunch does is very much suited towards using Bash. Trying to retrofit a Python layer in that calls out to various Bash scripts would cause a lot of headaches and a full rewrite would be better in that case, but a full rewrite of SteamTinkerLaunch is not something I want to think about right now :sweat_smile: I am still new as maintainer and don't want to go stripping SteamTinkerLaunch apart just yet.

In future, and this is pie-in-the-sky future we're talking about, I would like to break SteamTinkerLaunch down into several smaller scripts and decouple it a little at a time; "breaking down the monolith" one might say. But ignoring the complexity of some of what SteamTinkerLaunch does, given the sheer size alone it would be a lot of work. I would honestly say if I was going to do it, I would budget over a year before it's fully done once I go down that road. And there are too many other features in the pipeline to think about doing it just yet.

The reason I was thinking wxPython was the tabbed window interface for grouping relative settings apart from the whole?

Yad actually supports this, but not on Wayland. SteamTinkerLaunch used to have tabs but they were removed because Yad does not have support for "notebooks" as they're called under Wayland. And trying to tell when a user is on Wayland vs X11 and changing how the UI is shown, would be a real pain to do. I'm also not sure if wxPython has native Wayland support, but that's a much more minor issue since it would just result in some blur on HiDPI displays. When trying to use notebooks with Yad on Wayland, they spawn as separate window processes and it's just a mess.

I am a Plasma Wayland user, originally by necessity to comfortably use fractional scaling and higher refresh rate displays, and now by choice for the noticeable performance gains in games. So Wayland support is held as a critical for SteamTinkerLaunch in-part selfishly because I use Wayland and I want SteamTinkerLaunch to work for me ;-)

Thought I'd ask though.

And just one more time, I'd like to say thank you for even thinking to offer. This kind of initiative is part of what keeps me motivated to work on SteamTinkerLaunch (even before I was maintainer!).

I will try my best to test the dev branch this weekend.

Actually, I had meant to post this earlier, but this branch is in master now as of a few hours ago! I bit the bullet and merged it. I had actually meant to leave a comment but my personal life caught up with me for a few hours and I didn't get a chance. I'm going to write a more formal message after this monster of a reply heh.

Thanks to both of you for the discussion here, it is much appreciated and something I genuinely love to see :+1:

sonic2kk commented 1 year ago

With #674 merged, the mo2-overhaul branch is now in master (6a60f8ced42ff35c0cab92e2d34bca6cf9604797). I tested my existing New Vegas and Oblivion mod installs, and they worked. Enabling and disabling mods also worked, and installing a new mod worked too. With that and the positive feedback I had already gotten, I felt comfortable merging the changes.

However, please feel free to continue testing the implementation. Just now you can test it master :-) I will keep this issue open probably until SteamTinkerLaunch v12 releases, which should hopefully be very soon pending some further testing! I plan to open an issue to detail the final few steps needed before v12.

Thanks everyone :-)

har-nick commented 1 year ago

I did forget to mention, but I've also created a full SSE mod environment with xEdit, Nemesis, Synthesis, etc, and besides some minor graphical issues, it's all worked flawlessly.

I've one issue, but I'm unsure it's related to the overhaul branch specifically, so I'll make a new report.

I'm glad to see the overhaul branch has been pushed. A lot of work was put into it, and I'm sure a lot of users are grateful for it, so thanks!

sonic2kk commented 1 year ago

Wow, that is a ringing endorsement that it pretty much works as intended! Lots of work was indeed put into it, I helped out with a little bit of research around it and Frostworx did the actual legwork programming on it.

Small tip that may help with the graphical issues: using Winecfg you can set the Wine theme to (none) and this tends to help with issues like black boxes, though at the expense of GUI programs looking a bit Windows-2000-y. For MO2 itself you can set a custom theme from its settings to get around this, but other programs may not have such luxuries.

And thank you for opening an issue about the problem you were facing :-)

tetoNidan commented 1 year ago

sonic2kk, I totally understand. A re write isn't something you want to get into at the moment. I dont even use windows so there is no point in me heading a project like that, especially when I'm not familiar with the code base at all.

However if, when you do want to do a rewrite I can be pretty handy in python if you ever do need. My bash is rubbish though! I have done allot of work in python.subprocess and python.psutil. Written a few complex ffmpeg wrappers a few times for encoding multiple videos in different formats and resolutions for web development. with and without real time TTY output from the subprocess. I also have experience with Kivy but none with QT.

If or when the time comes, keep me in your wheelhouse, i'd like to give back.

EDIT: Forgot to mention I have experience with pytinstaller single file executable's compiling on windows, mac and linux compiling from a single source code.

Thanks again!

frostworx commented 1 year ago

Hi all, great discussion. Good to see new people interested in helping in various ways 👍

If I knew initially that steamtinkerlaunch would become so huge, I would have started directly with writing it in c++ or python. Writing it in bash to its current scope already was (almost) insane, and rewriting it now from scratch would be as well :) Getting rid pf all yad/bash barriers the project has currently is very tempting of course, so whenever someone starts an attempt good luck and much fun!

Instead of contributing to steamtinkerlaunch directly, enhancing yad by contributing functions we require in steamtinkerlaunch might be a great idea. For example, a long time ago steamtinkerlaunch had tab menus, which helped a lot for a clear overview. Unfortunately the implementation used in yad for tabs was (is?) not wayland compatible (required xembed) and therefore I rewrote the whole gui part.

For sure you will have a better readability when splitting steamtinkerlaunch into multiple parts, but you should seriously consider if it is worth the work and time it requires. It would not only require the time for rewriting this, but also for testing every single function and their interoperability and finally the split out parts need to stay available from the SLR sandbox, which might make maintaining the rewrite even harder in the future.

sonic2kk commented 1 year ago

but you should seriously consider if it is worth the work and time it requires. It would not only require the time for rewriting this, but also for testing every single function and their interoperability and finally the split out parts need to stay available from the SLR sandbox, which might make maintaining the rewrite even harder in the future.

This is absolutely one of the major concerns :-) And yet another reason why it hasn't even been fully planned. It's just one of those things that I keep in my mind but that I have very little motivation to plan out. Even Proton's proton script is just a behemoth Python script.

Something I would rather do instead is "document" more of the code that already exists. Some big parts of the codebase like MO2 and Vortex have comments around where they begin and end, and I did something similar for HMM support. I think I'll try to do more of this going forward around the code where possible :-)

sonic2kk commented 1 year ago

Thanks to every user who participated here in helping with the development of the MO2 overhaul. There is still an outstanding issue that I'm looking into (#682) and other users who have issues are encouraged to file separate bug reports. This issue should no longer be required, so I will be closing it and unpinning it :-)

Thanks to all once again :partying_face: Happy tinkering!