Resolve issues with JRuby plugin system

[garyttierney]

All in all, the reception of JRuby in Apollo hasn't been great. It has problems which make it hard for editors to help beginners write code, and generally isn't a well liked language.

Replacing Ruby with Groovy would allow plugin developers to achieve the same end result with their plugins using the same API, but with clearer code and much better semantic analysis during development. Additionally, it is possible to debug and step through an embedded Groovy plugin using most popular IDEs, with JRuby it is not.

[cubeee]

Ideally something quasar compatible

[Major-]

Not opposed per se but does Groovy actually solve the problem ("make it hard for editors to help beginners write code, and generally isn't a well liked language.")?

[Lmctruck30]

I would have to say for the debugging I would also like this. I personally removed ruby because having to constantly restart to test code made it a pain.

[garyttierney]

@Major-

I'm not 100% set on Groovy, however, it's the first JVM based language that comes to mind when considering something like Java. Kotlin looks like it could also be a good fit. Really, the first problem is solved by using a language that is interoperable with Java.

A big problem with Ruby (for editor tooling), is the lack of type information to accomplish any meaningful static analysis. If we could somehow create type-hints across the plugin API then I think a lot of problems would go away. Though, you're still left with the inability to step through your code.

Using Groovy solves the type information problem by allowing for type-specifiers. It is also syntactically similar to both Java and Ruby in some aspects. If plugins are also executed using the GroovyShell API, then it should be completely possible to step through them from an IDE.

There's a small write up by the Groovy folks on static-typing in a dynamic language which might be worth a read: A static theme for a dynamic language A static theme for a dynamic language

[Major-]

Will support either Groovy or Kotlin, provided we can still create a nice plugin API. Replacing the dialogue system, or even just the message hooking, in a similarly-pleasant fashion might prove impossible in other languages.

[garyttierney]

Another issue which is perhaps relevant here is deferred execution of plugins. Currently lists of scripts must be manually maintained, because script1.rb can use declarations from script2.rb. Separating declarations / definitions from code that is supposed to interface with the 'bootstrap' plugin could solve that I suppose (i.e., parse all scripts into Plugin objects first, and then do plugins.forEach(Plugin::run)).

I think this is a QoL improvement for the most part though.

[garyttierney]

@Major-

Picking up on what you said above, the dialogue plugin seems like it would be a good benchmark for language flexibility. I'll work on porting some examples over to see how well they fit.

Getting Quasar into scripting would also be pretty amazing, since it would create an extremely fluent plugin API for asynchronous actions.

[frostbit3]

What about Python? It would be really easy to make a clear and simple DSL.

[cubeee]

Python and Ruby pretty much swim in the same waters and offer no real benefit over the other so I'd rather stay with JRuby than consider Python

[garyttierney]

NACK on Python. Jython suffers from exactly the same problems as JRuby tooling wise, and with regards to syntax, I'm sure a lot of people would rather stick with Ruby.

[cadamsdev]

I would recommend Kotlin over Groovy. 100% interoperability with Java. Essentially a far less verbose and improved version of Java. You can pickup in no time if you already know Java.

[Detailed]

@cubeee Yes, Python and Ruby are similar, however I would argue that Python is more popular than Ruby. You would be able to get more users contributing if there is support for both Python and Ruby.

Especially in my case, I have been using Python for years now and Ruby hardly looked at.

Basically I like the idea of multiple language support

[thestupidity]

I've had better experience working with Groovy than Kotlin.

On Sun, Jan 8, 2017 at 3:54 PM, Marcello notifications@github.com wrote:

@cubeee https://github.com/cubeee Yes, Python and Ruby are similar, however I would argue that Python is more popular than Ruby. You would be able to get more users contributing if there is support for both Python and Ruby.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apollo-rsps/apollo/issues/310#issuecomment-271178701, or mute the thread https://github.com/notifications/unsubscribe-auth/AHqNtBnIIbIZF5QHFgjhkTs_U78WIQuAks5rQU0BgaJpZM4LY126 .

[garyttierney]

For what it's worth, supporting multiple scripting languages upstream is a non-starter.

[thestupidity]

The argument shouldn't be what syntax is easier for people to understand and use. What language can better benefit the project is what is important.

[khkramer]

The ability to write scripts in Ruby was actually a huge selling point for me to use Apollo for my rsps development.

It doesn't matter what scripting language they'll pick, there will always be people who aren't happy with it.

I for one think the developers are right in focusing on one scripting language and think Ruby is a very solid choice for its readability and meta-programming (DSL) possibilities.

[rmcmk]

Kotlin has just released 1.1 which has support for native coroutines and headless scripting (perfect for the design of a RuneScape server), may be worth looking into.

[garyttierney]

@ryleykimmel currently exploring Kotlin as an option. I have to admit that it's really nice, and fits the use case quite well. I've yet to look into where we can apply usage of continuations.

I've also been exploring ways for distributing plugins:

22:18 < sfix> grahamedgecombe: did you ever think about methods for packaging and distributing plugins?
22:19 < sfix> exploring kotlin as a choice for a scripting language atm and to get it playing nicely i had to give the plugin a buildscript with a dependency on :game
22:19 < sfix> made me think about ways to leverage existing tools
22:22 < grahamedgecombe> no, I didn't
22:23 < grahamedgecombe> dep on :game seems reasonable
22:23 < sfix> yes having plugins as modules with their own build scripts does too
22:23 < sfix> plugin tests!
22:23 < grahamedgecombe> + for third-party stuff could upload apollo to maven central
22:23 < grahamedgecombe> and then they can depend on it
22:24 < grahamedgecombe> that kinda relies on having stable releases first though :p
22:24 < sfix> i don't expect to see third party plugins before a stable release :p
22:25 < sfix> i think i'll explore this a bit more
22:25 < sfix> definitely an attractive option
22:26 < sfix> solves the plugin dependency resolution problem too (so apollo doesnt need to take care of it)
22:26 < sfix> plugins can just have dependencies like project(':core-plugin:attributes')
22:28 < sfix> hmm with something like https://wiki.eclipse.org/Aether/What_Is_Aether this would work pretty well

[frostbit3]

Whats the status on this? I'm definitely in agreement with using Kotlin as the replacement.

[garyttierney]

@frostbit3

Nothing has really been done to move it forward. If you'd like to get involved, the first thing is laying groundwork for a bootstrap.rb port.

@ryleykimmel

I'd quite like to use script templates for plugins, so we can just load Gradle like plugin scripts which get instantiated as some org.apollo.game.plugin.Plugin object. I've seen you comment on the Kotlin issue tracking this, so wondering if you have any ideas on how we can best implement that.

Something which might be noteworthy here: being able to step through plugin code is a HUGE advantage. If we end up with Kotlin but no debugging, I'd be wary of going down the route of replacing the scripting language at all.

[Major-]

Nothing has really been done to move it forward. If you'd like to get involved, the first thing is laying groundwork for a bootstrap.rb port.

I'd still like to see a dialogue plugin replacement in [language]

[rlgenesis]

I personally do not agree with the change in the plugin system from Ruby to any other language. Ruby itself is a solid language and I feel that it has very little to do with the success of Apollo (in reference to people not picking it up). From my understanding Apollo was meant to be a framework where you can easily implement and share content through a repo-like system, but by not providing any sort of documentation on how to use the core library, it actually hinders the speed in which someone can actually add content.

When I first picked up Apollo the first thing I immediately noticed is that I would have to go through the Java source code in order to add something as simple as Woodcutting through Ruby as there was no documentation, and very little to reference on how to create world objects. So the whole "don't touch the core, and just work on scripting" is rendered ineffective when you have to find yourself going through it regardless just to start writing content.

Besides the little documentation there are also a few major bugs that need to be sorted that actually prevent you from adding newer content. The region one is pretty major, and there are still 39 other unresolved issues some from as far as 2 years ago which it seems no one is willing to work on. I feel that the core issues need to be resolved first before considering changing to an entirely different language.

My opinion on Groovy in general is that it will receive the same reception as Ruby, Python, or any other language. If there is no documentation, and a lot of issues with the core, it'll hinder content developers from working effectively unless they know Apollo front and back.

[laxika]

What about supporting more than one language? With a good abstraction layer it can be done.

[garyttierney]

What about supporting more than one language? With a good abstraction layer it can be done.

Not happening. Our current abstraction layer lets you do this, but we'll never support several languages upstream.

[garyttierney]

I personally do not agree with the change in the plugin system from Ruby to any other language. Ruby itself is a solid language and I feel that it has very little to do with the success of Apollo (in reference to people not picking it up).

This is purely anecdotal. There are very real problems with how we're using JRuby. Some of them have been outlined above.

When I first picked up Apollo the first thing I immediately noticed is that I would have to go through the Java source code in order to add something as simple as Woodcutting through Ruby as there was no documentation, and very little to reference on how to create world objects.

Right, we need documentation. Create a separate issue for this. I don't feel like this is related to the fact that it's impossible to step through embedded JRuby code in a debugger.

Besides the little documentation there are also a few major bugs that need to be sorted that actually prevent you from adding newer content. The region one is pretty major, and there are still 39 other unresolved issues some from as far as 2 years ago which it seems no one is willing to work on. I feel that the core issues need to be resolved first before considering changing to an entirely different language.

Patches are welcome.

My opinion on Groovy in general is that it will receive the same reception as Ruby, Python, or any other language. If there is no documentation, and a lot of issues with the core, it'll hinder content developers from working effectively unless they know Apollo front and back.

Involving Groovy is just a consequence of the problems we've faced with JRuby. The point isn't to switch to another language, the point is to fix the problems we're encountering currently.

[Major-]

@laxika

What about supporting more than one language? With a good abstraction layer it can be done.

Main problem with this (aside from the obvious complexity) is interoperation between the languages

[garyttierney]

To clarify the current problems we're experiencing with how the Apollo plugin system is built:

• Impossible to debug plugins (most important IMO)
• Dependency resolution / deferred execution (see https://github.com/apollo-rsps/apollo/issues/310#issuecomment-270007004)
• Testing (no way to currently write tests around plugins)
• Packaging (as mentioned above the idea is for a plugin repository sort of setup, this currently isn't supported)

Some of these problems could be resolved using our current JRuby plugin system, while some can't (try to debug an interpreted JRuby script).

Some additional UX issues which a change to Kotlin/Groovy fixes:

• Static analysis (aka editor support)
• Easy packaging integration (package as maven artifacts, use a nexus for apollo plugins)

[garyttierney]

Note: the title has been updated, but that doesn't rule out the issues being fixed by replacing Ruby.

[Major-]

Looked into kotlin a bit and I'm pretty sure its the best alternative to ruby in terms of replacing existing cool plugin stuff (e.g. dialogue) nicely, as well as solving the problems listed above.

Only plus for groovy I can think of is that its what gradle uses

[rmcmk]

@Major- Gradle has semi-recently released Gradle Script Kotlin

[garyttierney]

@ryleykimmel I believe Gradle uses Kotlin ScriptTemplateDefinitions to go about this. Not sure how we go about the same for Apollo, but I do know that it's landed (or is at least available) in 1.1

[rmcmk]

When I was tinkering with this @garyttierney I either needed to write my own template definition that would work for my use case and resolve my dependencies or wait for Kotlin to support JSR-223 which they have done as of 1.0. You can use a simple ScriptManager wrapper for dealing with Kotlin scripts just like we do with Ruby at the moment.

[garyttierney]

@ryleykimmel Hmm, do you have an example of that? I'm not sure how template definitions fit in with JSR-223.

[rmcmk]

IIRC they are not required when using JSR-223

[Major-]

Adding to stable release milestone because one shouldn't be made without resolving this question

[Major-]

Currently we're looking into kotlin, using kotlin scripts. DSL very much a WIP but here's a sample:

/**
 * Hook into the [ObjectActionMessage] and listen for when a bank booth's second action 
 * ("Open Bank") is selected.
 */
on { ObjectActionMessage::class }
    .where { option == 2 && id == BANK_BOOTH_ID }
    .then { BankAction.start(this, it, position) }

• on, where, and then are all functions that take lambdas - you can drop the parens if you're only passing a lambda, just like in ruby.
• where and then and are evaluated from the perspective of the Message object, so option and id are referring to ObjectActionMessage.option and ObjectActionMessage.id.
• it is a special identifier in kotlin that can be used in any lambda when it only has one parameter, to avoid having to specify the argument; then could be written equivalently as:

.then { player -> BankAction.start(this, player, position) }

Here's the full plugin:

package org.apollo.game.plugin.impl

import org.apollo.game.action.DistancedAction
import org.apollo.game.message.impl.NpcActionMessage
import org.apollo.game.message.impl.ObjectActionMessage
import org.apollo.game.model.Position
import org.apollo.game.model.entity.Npc
import org.apollo.game.model.entity.Player
import org.apollo.game.model.inter.bank.BankUtils
import org.apollo.net.message.Message

val BANK_BOOTH_ID = 2213

/**
 * Hook into the [ObjectActionMessage] and listen for when a bank booth's second action
 * ("Open Bank") is selected.
 */
on { ObjectActionMessage::class }
    .where { option == 2 && id == BANK_BOOTH_ID }
    .then { BankAction.start(this, it, position) }

/**
 * Hook into the [NpcActionMessage] and listen for when a banker's second action
 * ("Open Bank") is selected.
 */
on { NpcActionMessage::class }
    .where { option == 2 }
    .then {
        val npc: Npc = world.npcRepository.get(index)

        if (npc.id in BANKER_NPCS) {
            BankAction.start(this, it, npc.position)
        }
    }

/**
 * The ids of all banker [Npcs][Npc].
 */
val BANKER_NPCS = setOf(166, 494, 495, 496, 497, 498, 499, 1036, 1360, 1702, 2163,
                        2164, 2354, 2355, 2568, 2569, 2570)

/**
 * A [DistancedAction] that opens a [Player]'s bank when they get close enough to a booth 
 * or banker.
 *
 * @property position The [Position] of the booth/[Npc].
 */
class BankAction(player: Player, position: Position) : 
    DistancedAction<Player>(0, true, player, position, DISTANCE) {

    companion object {

        /**
         * The distance threshold that must be reached before the bank interface is opened.
         */
        const val DISTANCE = 1

        /**
         * Starts a [BankAction] for the specified [Player], terminating the [Message].
         */
        fun start(message: Message, player: Player, position: Position) {
            player.startAction(BankAction(player, position))
            message.terminate()
        }

    }

    override fun executeAction() {
        mob.turnTo(position)
        BankUtils.openBank(mob)
        stop()
    }

    override fun equals(other: Any?): Boolean {
        return other is BankAction && position == other.position
    }

    override fun hashCode(): Int {
        return position.hashCode()
    }

}

There's still a good amount of work to do (mostly testing-related stuff) but kotlin looks promising.

[rmcmk]

Can you commit this stuff somewhere (I see the "kotlin-experiments" branch) and make some issues as to what you're trying to solve? Would be keen to help out. @Major- @garyttierney

[Major-]

As an update: barring any serious unforeseen problems we are likely to be moving all plugins to kotlin, and removing support for ruby plugins in the near-ish future. When this happens we will convert and replace all ruby in one big change (at least on the master branch), so there will not be any period of half-ruby half-kotlin.

Kotlin brings a good number of benefits: aside from the language itself, #337 contains discussion on improving plugin packaging and distribution. We'll also be revamping the test architecture to ensure plugins can be properly tested (and even disregarding plugins, apollo's current coverage is abysmal) and introducing improved plugin DSLs (our current command DSL leaves a lot to be desired, for example).

[garyttierney]

Major milestone no. 1 for a new scripting language: debugging https://i.imgur.com/OKESxMR.jpg

[garyttierney]

Build tool and IDE integration is superb: https://streamable.com/copi9. Next up is testing.

[garyttierney]

@Major- @ryleykimmel food for thought on coroutines https://www.slideshare.net/naughty_dog/statebased-scripting-in-uncharted-2-among-thieves

[garyttierney]

9353daabc3c087bb119d779d8be6fcfca998b6e0 introduces support for testing as well as incremental compilation. Now if a plugin has tests that fail, the build will fail as well. The next thing that is needed is a cohesive test framework built for testing plugins (load a plugin with a mock of the world and fire its message handlers, then assert the results). A good place to start with would be the 'bank' plugin.

[garyttierney]

A further update: we now have a much better testing framework (which keeps getting better), seen here: https://github.com/apollo-rsps/apollo/blob/kotlin-experiments/game/src/plugins/dummy/test/TrainingDummyTest.kt.

Along with that, the plugin compiler code was refactored into a gradle buildSrc project to speed up compilation times. All plugins will now be compiled within the same JVM instance as opposed to forking a new instance for each plugin.

[garyttierney]

Following on from the initial discussion, I've had time to evaluate the place for continuations in plugin code. The first step is tying this into Actions and allowing developers to do a non-blocking wait until the next execution. This looks quite nice:

fun action() : ActionBlock = {
    player.sendMessage("You drink the potion")
    wait(pulses = 1)
    player.sendMessage("It heals some health")
}

and allows us to do away with the many small state machines that we create to keep track of action state (i.e., in the above example we'd check if the action had previously been started, and if so, show the second message).

The wait function internally is a suspendable function that queues a coroutine with the continuation created for the original action block:

    private suspend fun awaitCondition(condition: ActionCoroutineCondition) {
        return suspendCoroutineOrReturn { cont ->
            next.compareAndSet(null, ActionCoroutineStep(condition, cont))
            COROUTINE_SUSPENDED
        }
    }

    /**
     * Wait `pulses` game updates before resuming this continuation.
     */
    suspend fun wait(pulses: Int = 1) = awaitCondition(AwaitPulses(pulses))

However, instead of dispatching these coroutines to an executor they are held in memory along with a condition that checks if they should be resumed until the Action is pulse()'d again.

    private fun resumeContinuation(continuation: Continuation<Unit>, allowCancellation: Boolean = true) {
        try {
            continuation.resume(Unit)
        } catch (ex: CancellationException) {
            if (!allowCancellation) {
                throw ex
            }
        }
    }

    /**
     * The next `step` in this `ActionCoroutine` saved as a resume point.
     */
    private var next = AtomicReference<ActionCoroutineStep>()

    /**
     * Update this continuation and check if the condition for the next step to be resumed is satisfied.
     */
    fun pulse() {
        val nextStep = next.getAndSet(null)
        if (nextStep == null) {
            return
        }

        val condition = nextStep.condition
        val continuation = nextStep.continuation

        if (condition.resume()) {
            resumeContinuation(continuation)
        } else {
            next.compareAndSet(null, nextStep)
        }
    }

If the condition is satisfied, we resume the continuation and jump straight back into where we left off previously in the action. The example given is simple (waiting 'til the next execution), but this allows is to wait on any condition that can be checked every pulse without blocking any other game logic, which is a big win.

There are still some bigger use cases to solve down the line, perhaps for modelling more complex actions like combat, but this is a good general implementation that fits most of our simple usecases at the moment.

The initial implementation of this code can be found here: https://github.com/apollo-rsps/apollo/blob/kotlin-experiments/game/src/main/kotlin/org/apollo/game/action/ActionCoroutine.kt

[Major-]

Closing as we decided to move to kotlin a long time ago. See #338, #361 for progress.