Renpy Code Formatter

[linkb15]

I do believe we need a code formatter in Renpy somehow

[duckdoom4]

What kind of formatting are you looking for? Code formatting can probably be solved by installing a second extension.

[linkb15]

code formatter like Black for python

[LuqueDaniel]

Extensions such as Prettier allow you to create plugins to support other languages using the Prettier API. I think implementing a formatter in vscode-language-renpy would require implementing things that are already perfectly covered by Prettier. In this way, it could be used independently of this extension or even vscode.

[linkb15]

anyone has tried using prettier for renpy?

[duckdoom4]

What kind of formatting would you like to achieve specifically? If it's something small and not too difficult to implement (and benefits everyone), I might try to implement it.

Even though I agree with @LuqueDaniel and I also think this should probably be independent.

[LuqueDaniel]

@JelleInfinity Good point 👍 , the scope of the feature.

[linkb15]

I never used prettier with python before.. but I have tried using prettier with JavaScript projects.. so if possible we can fix indentation and spacing between variables and functions in general. not sure how small it is though..

[duckdoom4]

I do have it setup in our current development workspace for typescript, json and python files. Though to be completely honest I am not sure if it actually works on python files. We only have one of those and it never changes.

[duckdoom4]

I suppose you mainly want this to function within the python scopes? Maybe it's possible to apply a formatter to a scope?

Auto indentation should work, if it doesn't then I will push a fix on the main branch. (Note that this is not fixing already messed up indentation, but indenting when pressing enter after the ' : ' symbol).

Spacing and indentation is not super difficult to do, I might look into that. But it would be way on the backlog as I think other extension could be used for this. I'm currently working on improving the syntax highlighting in the new branch.

[linkb15]

nice.. working on python3 syntax highlighting right..

yea i want the feature for renpy globally on screen language for example..

ah okay .. if it is in the backlog..

good luck finishing the other features man 👍

any other extension as workaround you recommend and how to use them with renpy?

I can try to help and make PR with this too if u are willing to onboard a bit on the project.. though I havent made any vscode extension previously

[duckdoom4]

I still recommend prettier if it works.

You can look through this list for all available formatters. (Please let us know if you found a nice/working one. Then we can add it to the recommended extensions list)

Or you can use this to setup a custom script.

And yes, that is for python3 syntax, but also a general rewrite of the syntax highlighting feature. You can view the progress here

[LuqueDaniel]

@linkb15 I'll be happy to help you with any PR :)

[duckdoom4]

Ah I didn't read your edit.

Any help is appreciated :). You can find a bit of documentation here.

As for how to implement it, try to keep it as simple as possible. When the new tokenizer is finished you can use that to make more complex logic.

You probably need to search through the document for patterns that match something that you think should be formatted. Like for example double spaces. You can use https://regexr.com/ to create the patterns, that's what I always use.

Feel free to ask me if you get stuck

[linkb15]

Hi @JelleInfinity I do get stuck trying out simple vs code extension development following this guide. https://code.visualstudio.com/api/get-started/your-first-extension

where I could not even do the F5 which my vscode does not recognize the extensionHost as the debugger.. not sure what happened.. after searching around, it says that the extensionHost is built-in vs code but seems not the case for my local :(

[duckdoom4]

Not sure what happens there, but it took me a bit to figure out how to get it to work too. I should add a 'how to' to the readme.

Anyway, just forget about the whole yo generator thing. You only need that if you want to create a completely new extension (from scratch).

Just make sure you installed the latest vscode version and have Node.js installed. Also test to see if you can run npm from cmd.

Then you just open vscode and select open folder. Pick the cloned repo folder, making sure package.json is now in the root of your workspace. After that you should be able to press f5 to launch a second window with the extension enabled

[FaceCrap]

I wish there was a simple way to do this, I've been wanting a formatter for a while now after having been drawn into assisting in VN code cleanup/maintenance, but adding a formatter using the above mentioned options looks even more daunting than trying to write an extension. Why on earth would I need to install node.js and god knows what overhead to use a thing called Prettier? I tried, but I just have zero experience with this thing and it looks to be needing a boatload of configuring before it even will do anything with renpy files... I got as far as adding the Prettier extension in vscode, adding a document selector, but then it begins complaining about overrides because it don't know what to do with rpy files... and where the heck do I specify those overrides and where to get a parser that knows renpy format...???

Honestly, I for one, given the choice, would rather have this formatting embedded in this extension, but it looks like that is not going to happen anytime soon.

So, effectively I'll be "forced" to look into Prettier, but I would very much appreciate it if anyone has already done this, to share what is needed and explain what I need to do to get this working on rpy files, preferably without the need to install god knows what in extra software (like please no node.js) and maybe even have a parser?

[duckdoom4]

Thanks for the heads-up on the difficulties you faced with trying to setup prettier for renpy files.

That gives us a bigger reason to try and add some form of formatting. (Be it an example on how to add it to an already existing extension, or adding it to this one)

Could you provide some examples of what formatting issues you're currently facing in your project? That would help us understand the kinds of problems we should aim to fix.

All that said, it's still in the backlog. When we have time to continue development, the first thing we'll be working on would be semantic highlighting

[FaceCrap]

The thing is that Prettier needs a parser to do the actual work, or so it seems. And I've no idea how to go about to create one seeing as I couldn't find an existing one. One big disadvantage I noticed is that it requires configuring on a per project basis... which is cumbersome to say the least.

Basically the biggest issue I have with formatting is proper indentation. For newly created files that isn't such a big deal, but when you get presented existing code files it's a whole different kettle of fish. These sometimes have almost no other formatting than indented "if" and "menu" blocks, and "label" blocks not being indented at all, or using way more than the default 4 spaces when indent is actually used. The lack of proper formatting makes it hard to use folding as you're never sure if parts of code getting folded are actually part of the folded block.

The thing I'd most like to be able to have automated for now is to have a blank line inserted whenever the indent is reduced, e.g. when a menu choice block ends and the next choice is encountered having a blank line before the next menu choice item, and after the last choice block, same for "if/elif/else" branches/blocks. Preferably with a command, or automatically on load/save I've even thought about just creating a simple extension for that because I suspect there's got to be something keeping track of indents seeing that it can show guidelines at indent intervals, but at the moment I don't know enough about VSCode's internals to be able to do that, plus it seems I then too need to install things like Node.js, which I'm reluctant to do since that would present an additional learning curve.

[duckdoom4]

We have a parser, it's on the dev branch :)

[duckdoom4]

Blank lines and spaces are rather easy to add and I think that would be a good addition to this extension. So can definitely look into adding this.

About the node.js part;

Node.js is a JavaScript runtime that allows desktop apps to run JavaScript. Think of it like 'installing JavaScript' like you would 'install python'. The things you'd need to learn is how to use Typescript, not much else. (Besides the vscode API ofc)

[FaceCrap]

We have a parser, it's on the dev branch :)

As in: to be used with Prettier? Seeing a lot of .ts files in that branch I take it that in order for me to try it out I'll have to somehow create a build? Because I've no idea how to do that... And even if it gets to prerelease state... then what? I assume you'll include instructions to make Prettier use it? (if it's intended to be used that way...)

[LuqueDaniel]

From the parser in the development branch, it's possible to develop the necessary functionality to create a formatter. But not that it's ready to be used by Prettier.

[linkb15]

https://pypi.org/project/renpyfmt/1.0.0/

found this, preliminary test works for inline python code.

input in ./game/script.rpy: $ name =renpy.input("What's your name?") I removed the space after the equal sign here

run in terminal: renpyfmt ./game/script.rpy ./game/script.rpy

output in ./game/script.rpy : $ name = renpy.input("What's your name?") the space is added here.

https://github.com/LuqueDaniel/vscode-language-renpy/assets/24437987/540211a0-07d1-4d29-a514-92e55145244d

[duckdoom4]

Oh, that's really nice. We could build that in if the author is okay with that

[kobaltcore]

Heyho, creator of renpyfmt here 👋

I was made aware of this issue today and would certainly like to participate in driving this particular part of the ecosystem forward. First things first: I certainly wouldn't mind if my tool were to be integrated into this extension (or anything else for that matter, feel free to use it for whatever you like), so that won't be a roadblock from my end.

By the way: This doesn't just work for inline Python, it also works for python early, python and init python blocks.

For now, renpyfmt only formats Python inside Ren'Py script files since that was reasonably easy to implement. However, my final goal for this project is to properly format Ren'Py script itself as well. I've already got some work going on another branch which consists of pulling apart Ren'Py and extracting the AST-related portions to then use that as a way of generating clean, properly-formatted Ren'Py script, standalone. black is my main inspiration for this tool: The goal is to eventually arrive at one source-of-truth 'spec' for how each part of the syntax is to be formatted and to uphold that as the main way to format Ren'Py in all cases, though the planning on that is flexible of course.

I'm not the fastest with working on OSS due to my day-job already taking up a lot of my time, but I do intend to keep working on this as well as providing support for the cases renpyfmt currently fails. Building up a test suite of formatting cases to validate things on is also on my todo list still.

There are a couple things that might make integration iffy until they're fully resolved:

• It's only been tested locally by me, on scripts I have available. Contact with the whole wide world is a big difference to that of course, so there may be some hidden bugs lurking about.
• The interface is not yet stable: While the general I/O works, I don't yet know if further expansion of the tools abilities will necessitate API changes. Those will be semver'd of course, so shouldn't be that big a deal, just wanted to mention it.
• It's currently not configurable, this is planned for later. There will probably need to be some kind of renpyfmt.toml config file at some point to configure black and renpyfmt itself.

There are also two issues with Ren'Py script itself that may end up being very difficult to handle:

1. Comments are stripped while parsing, meaning they don't end up in the AST. They're relatively easy to add in though, so I may make a PR to Ren'Py proper at some point to get them into the AST and thus make it easier to consider them as well when formatting.
2. Control flow is currently not fully decidable. Here's a quick example:

   
   label A:
   "dialogue 1"
   "dialogue 2"

label B: "dialogue 3" "dialogue 4"

The AST will properly detect dialogue 1/2 as children of label `A`, but this is not the case for label `B`: Its children will be empty and dialogue 3/4 will exist as top-level nodes right after it. This is fine for Ren'Py itself because it simply proceeds to the next node while executing, but it, too, doesn't know they should actually belong to label `B`.

Unfortunately I haven't yet found a way to detect whether successive nodes are actually supposed to be children of the preceding node (if that node can have children) or not. As such, formatting in such a case can go wildly awry. Figuring that part out will largely unlock our full ability to format Ren'Py script, as far as I can tell right now.

[CaptainCaption]

The new official version of the Ren'Py extension is severely lacking in keywords for onEnterRules' indentations. I was able to create a very messy fix on my end by adding commonly indented ATL and displayable keywords to the beforeText regex string (i.e., "parallel" and "show" ), but this is obviously not a sufficient solution. While lacking richer support of later releases, the 2.0.16 version of LuqueDaniel's unofficial Ren'Py extension was the most recent release which kept (to my memory) flawless indentation and formatting.

I personally subscribe to the idea that syntax rules during coding are more helpful than a full file formatter, so as a relevant aside, I've used the Python Paste and Indent extension with good results as it understands Python indentation better than most formatters and works around VSC's very poor indentation of Python files, but something about this version of the Ren'Py extension creates a very heavy CPU load that creates up to 10 seconds of delay between keypress and paste; furthermore, when profiled with the Developer: Show Running Extensions command, Ren'Py profiles so slowly that it is flagged as an unresponsive performance issue. I can supply the saved extension profiling and put this in a new issue request, but this was not an issue on any of the previous releases and I have far from a poor computer.

[duckdoom4]

@CaptainCaption

Hi thanks for the bug report. I don't think the previous version did those keywords correctly. I just added syntax highlighting to those keywords, the previous version didn't support those properly.

That said, I totally forgot about updating the indentation rule to support those missing block keywords. I'll add them when I find some time!

Also, the profiling file would be great to have. Please add it to a new issue :)