[FEATURE] Kotlin Multiplatform support

[mykola-dev]

Hey guys! Is there any reason why this lib is jvm only? I would like to use it in my KMM project (android, web, desktop, ios) How much effort is required to migrate the lib to KMM?

[christian-draeger]

Hey, I would definatly like to have the library support multiplattform. I think especially in the js/web world it could have great benefit. But so far I have zero experience with creating multiplatform libraries in kotlin. If someone would be willing to investigate and provide a pull request it would be highly appreciated.

Some of the modules are plain kotlin anyway or use libraries that are also multiplatform (like ktor client) and should thereby (I guess) be easily convertable to multi platform

[mykola-dev]

as i see the lib uses jsoup under the hood. so the main effort - migrate/rewrite it with KMP with preserving the public api.

[christian-draeger]

Following modules are pure kotlin:

• assertions
• base-fetcher
• dsl

Following modules have jvm specific dependencies which would need to be replaced on none jvm plattforms:

• html-parser (uses jsoup)
• • regarding JS it could maybe be possible to use document api directly?
• • on native maybe something like xsoup?
• http-fetcher (uses ktor client over apache-client)
• async-fetcher (uses ktor client over apache-client)
• browser-fetcher (uses html-unit to execute Javascript that is included in html)
• • on JS this should be possible somehow
• • on native don't know about an appropriate equivalent

Since I never wrote a KMM Lib and running short on time currently because of job and kid, would you mind to provide a PR @mykola-dev? This would really help 🙏

Maybe at least for 1 or 2 modules (if that is possible) to see how things work.

Another question would be how to publish it then.

[mykola-dev]

xsoup is a java lib too. i tried to migrate jsoup to kotlin (with further kotlin native migration), but it uses lot of jvm specific apis. estimated efforts are huge, so i abandoned this idea.

[aSemy]

I'm interested in making a PR for this. I've had a look at the code and I think some groundwork needs to be done first, as the Gradle config is applying the Kotlin/JVM plugin to all projects.

I can make a PR to create some convention plugins for the different Kotlin platforms, as well as other shared build config. The end result will be that the root build.gradle.kts file will be language-neutral, and the subprojects will have a Kotlin/JVM convention plugin applied.

I won't touch any of the source code, just the Gradle config.

[christian-draeger]

@aSemy this is awesome news. I really appreciate 💯🙏

[aSemy]

At first glance it looks like migrating to multiplatform will be pretty easy!

1. Make DomTreeElement a common class - different languages will have to implement their own versions. JVM can use JSoup, and as you noted, JS can use a native implementation
2. Ktor is multiplatform - so that shouldn't be hard to change
3. HtmlUnit is only used in one file. I expect that will be pretty easy to make multiplatform.

Something that might be difficult to change is that all the tests are using JUnit, which of course is JVM only. But migrating to something like Kotlin Test or Kotest should be reasonably easy.

[mykola-dev]

@aSemy how are you going to port the jsoup stuff to the native platforms?

[aSemy]

I'm planning to only keep the JVM target, but migrate the common Skrape code to commonMain. I'll make a commonMain expect class that any JVM native functionality will have to provide. So no, I'm not going to touch any of the JSoup library.

[Shabinder]

Would love support for this.

[christian-draeger]

Hey @Shabinder, Haven't been available for a while due to private responsibilities.

Would you still be open to help on this topic?