Closed lintool closed 8 years ago
Is this UDF ready to merge to master?
ExtractImageLinks.scala
seems good to me.
Fixed and merged commit 04d105ded330e116da09f93f34e375c8262cd2f9
Do we have an example script that calls this? If so, happy to write-up and include in docs.
This is the script I wrote for Ryan Deschamps
import org.warcbase.spark.matchbox._
import org.warcbase.spark.rdd.RecordRDD._
val links = RecordLoader.loadArchives("src/test/resources/warc/example.warc.gz", sc)
.keepValidPages()
.flatMap(r => ExtractImageLinks(r.getUrl, r.getContentString))
.countItems()
OK. Testing and will write up.
Put a stub in here, please feel free to tweak and refine: http://lintool.github.io/warcbase-docs/Spark-Image-Analysis/
Needed for a hackathon team: UDF for extracting image links.