search

locationtech / geotrellis

GeoTrellis is a geographic data processing engine for high performance applications.
http://geotrellis.io
Other
1.33k stars 360 forks source link

GeoTrellis dependencies version compatibility #3546

Closed Cloud-1384 closed 1 month ago

Cloud-1384 commented 1 month ago

I am worried about the dependencies that Geotrellis need. I am unable to figure out the version compatibility of GeoTrellis, Spark and SBT. I used sbt 1.10.0, geotrellis 3.7.0, spark 3.5.1. I am unable to download the dependencies when I run the compile sbt command.

pomadchin commented 1 month ago

hi @Cloud-1384 what is your workflow with SBT / GeoTrellis exactly?

Cloud-1384 commented 1 month ago

Dear sir/mam, Greetings of the day!

Thank you for reaching out to me for helping me in my project of testing HDFS for geospatial workload.

I have a .tif file in HDFS and want to view it using Geotrellis. I have installed a multi-node setup of HDFS and Spark (versions 3.5.1 and 3.2.0) and have successfully connected HDFS to Spark. I also installed Scala and sbt (version 1.10.0) on a RHEL server.

In the project directory, I created a build.sbt file with the following content:

name := "GeotrellisSparkHDFS"

version := "0.1"

scalaVersion := "2.12.18"

resolvers += "Maven Central" at "https://repo1.maven.org/maven2/" resolvers += "LocationTech" at "https://repo.locationtech.org/geotrellis/"

val geotrellisVersion = "3.7.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "3.5.1", "org.apache.spark" %% "spark-sql" % "3.5.1", "org.locationtech.geotrellis" %% "geotrellis-spark" % geotrellisVersion, "org.locationtech.geotrellis" %% "geotrellis-spark-core" % geotrellisVersion )

However, when I run sbt compile, I encounter the following error:

WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator (file:/home/sathvika/.sbt/boot/scala-2.12.19/org.scala-sbt/sbt/1.10.1/jline-terminal-3.24.1.jar) to constructor java.lang.ProcessBuilder$RedirectPipeImpl() WARNING: Please consider reporting this to the maintainers of org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [info] welcome to sbt 1.10.1 (Oracle Corporation Java 11.0.23) [info] loading project definition from /home/sathvika/geotrellis-spark-hdfs/project [info] loading settings for project geotrellis-spark-hdfs from build.sbt ... [info] set current project to GeotrellisSparkHDFS (in build file:/home/sathvika/geotrellis-spark-hdfs/) [success] Total time: 1 s, completed 30-Jul-2024, 8:30:12 AM [info] Updating geotrellissparkhdfs_2.12 [info] Resolved geotrellissparkhdfs_2.12 dependencies [warn] [warn] Note: Unresolved dependencies path: [error] sbt.librarymanagement.ResolveException: Error downloading org.locationtech.geotrellis:geotrellis-spark-core_2.12:3.7.1 [error] Not found [error] Not found [error] not found: /home/sathvika/.ivy2/localorg.locationtech.geotrellis/geotrellis-spark-core_2.12/3.7.1/ivys/ivy.xml [error] not found: https://repo1.maven.org/maven2/org/locationtech/geotrellis/geotrellis-spark-core_2.12/3.7.1/geotrellis-spark-core_2.12-3.7.1.pom [error] not found: https://repo.locationtech.org/geotrellis/org/locationtech/geotrellis/geotrellis-spark-core_2.12/3.7.1/geotrellis-spark-core_2.12-3.7.1.pom [error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:346) [error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$39(CoursierDependencyResolution.scala:315) [error] at scala.util.Either$LeftProjection.map(Either.scala:573) [error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:315) [error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60) [error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:60) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:142) [error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:74) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:144) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:131) [error] at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:220) [error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:169) [error] at sbt.Classpaths$.$anonfun$updateTask0$1(Defaults.scala:3867) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [error] at java.base/java.lang.Thread.run(Thread.java:834) [error] (update) sbt.librarymanagement.ResolveException: Error downloading org.locationtech.geotrellis:geotrellis-spark-core_2.12:3.7.1 [error] Not found [error] Not found [error] not found: /home/sathvika/.ivy2/localorg.locationtech.geotrellis/geotrellis-spark-core_2.12/3.7.1/ivys/ivy.xml [error] not found: https://repo1.maven.org/maven2/org/locationtech/geotrellis/geotrellis-spark-core_2.12/3.7.1/geotrellis-spark-core_2.12-3.7.1.pom [error] not found: https://repo.locationtech.org/geotrellis/org/locationtech/geotrellis/geotrellis-spark-core_2.12/3.7.1/geotrellis-spark-core_2.12-3.7.1.pom

It seems that Geotrellis dependencies are not being resolved. Can you help identify any mistakes or suggest possible solutions? Feel free to reach out if you have any further queries.

Thank you, Sathvika.

Message ID: @.***>

pomadchin commented 1 month ago

@Cloud-1384 the geotrellis-spark-core dependency is indeed non existent, it should be just geotrellis-spark

pomadchin commented 1 month ago

To be more clear, this should be enough:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "3.5.1",
  "org.apache.spark" %% "spark-sql" % "3.5.1",
  "org.locationtech.geotrellis" %% "geotrellis-spark" % geotrellisVersion,
)

And I don't think we ever published geotrellis-spark-core

pomadchin commented 1 month ago

Closing this one as a resolved issue! Please feel free to reopen it / create more issues!

Cloud-1384 commented 1 month ago

One more issue while working with hadoop.

my build.sbt file configuration is:

Compile / mainClass := Some("Try")

version := "0.1" scalaVersion := "2.12.18" resolvers ++= Seq( "Maven Central" at "https://repo1.maven.org/maven2/", "Geotrellis" at "https://repo.locationtech.org/geotrellis/" )

val geotrellisVersion = "3.7.1"

libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "3.5.1", "org.apache.spark" %% "spark-sql" % "3.5.1", "org.locationtech.geotrellis" %% "geotrellis-spark" % geotrellisVersion, // "org.locationtech.geotrellis" %% "geotrellis-hadoop" % geotrellisVersion, "org.locationtech.geotrellis" %% "geotrellis-raster" % geotrellisVersion, "org.locationtech.geotrellis" %% "geotrellis-vector" % geotrellisVersion, "org.locationtech.geotrellis" %% "geotrellis-proj4" % geotrellisVersion, // "org.locationtech.geotrellis" %% "geotrellis-geotiff" % geotrellisVersion, "org.apache.hadoop" % "hadoop-common" % "3.3.6", "org.apache.hadoop" % "hadoop-hdfs" % "3.3.6", "io.geotrellis" % "geotrellis-spark-hadoop" % geotrellisVersion )

I am unsure if "io.geotrellis" %% "geotrellis-spark-hadoop" % geotrellisVersion this is present or not.

I am getting the following error:

[success] Total time: 1 s, completed 03-Aug-2024, 9:47:42 PM [info] Updating geotrellis-spark-hdfs_2.12 [info] Resolved geotrellis-spark-hdfs_2.12 dependencies [warn] [warn] Note: Unresolved dependencies path: [error] sbt.librarymanagement.ResolveException: Error downloading io.geotrellis:geotrellis-spark-hadoop:3.7.1 [error] Not found [error] Not found [error] not found: /home/sathvika/.ivy2/localio.geotrellis/geotrellis-spark-hadoop/3.7.1/ivys/ivy.xml [error] not found: https://repo1.maven.org/maven2/io/geotrellis/geotrellis-spark-hadoop/3.7.1/geotrellis-spark-hadoop-3.7.1.pom [error] not found: https://repo.locationtech.org/geotrellis/io/geotrellis/geotrellis-spark-hadoop/3.7.1/geotrellis-spark-hadoop-3.7.1.pom [error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:346) [error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$39(CoursierDependencyResolution.scala:315) [error] at scala.util.Either$LeftProjection.map(Either.scala:573) [error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:315) [error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60) [error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:60) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:142) [error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:74) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:144) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:131) [error] at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:220) [error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:169) [error] at sbt.Classpaths$.$anonfun$updateTask0$1(Defaults.scala:3867) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [error] at java.base/java.lang.Thread.run(Thread.java:834) [error] (update) sbt.librarymanagement.ResolveException: Error downloading io.geotrellis:geotrellis-spark-hadoop:3.7.1 [error] Not found [error] Not found [error] not found: /home/sathvika/.ivy2/localio.geotrellis/geotrellis-spark-hadoop/3.7.1/ivys/ivy.xml [error] not found: https://repo1.maven.org/maven2/io/geotrellis/geotrellis-spark-hadoop/3.7.1/geotrellis-spark-hadoop-3.7.1.pom [error] not found: https://repo.locationtech.org/geotrellis/io/geotrellis/geotrellis-spark-hadoop/3.7.1/geotrellis-spark-hadoop-3.7.1.pom [error] Total time: 14 s, completed 03-Aug-2024, 9:47:56 PM @.***:~/geotrellis-spark-hdfs$

The code I want to run is:

import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileSystem, Path} import geotrellis.spark. import geotrellis.spark.io. import geotrellis.spark.io.hadoop._

object HDFSFileChecker { def main(args: Array[String]): Unit = { // Initialize Hadoop configuration val hadoopConf = new Configuration() hadoopConf.addResource(new Path("/home/sathvika/hadoop-3.3.6/etc/hadoop/core-site.xml")) hadoopConf.addResource(new Path("/home/sathvika/hadoop-3.3.6/etc/hadoop/hdfs-site.xml"))

// Create a FileSystem instance
val fs = FileSystem.get(hadoopConf)

// Define the HDFS path to the file
val filePath = new Path("hdfs://localhost:9000/map.tif")

// Check if the file exists
val fileExists = fs.exists(filePath)

// Print the result
if (fileExists) {
  println(s"File exists at: ${filePath.toString}")
} else {
  println(s"File does not exist at: ${filePath.toString}")
}

// Close the FileSystem instance
 fs.close()

} }

But in the official documentation it is mentioned that we can import these: import geotrellis.spark.io._

import geotrellis.spark.io.hadoop._ 2 packages but I am getting error

HDFS https://geotrellis.readthedocs.io/en/latest/guide/tile-backends.html#hdfs

Choose HDFS if: you want a simple setup and fast write speed or the ability to write and read from Microsoft Azure’s Blob storage.

The Hadoop Distributed File System https://hadoop.apache.org/ (HDFS), as the name implies, presents a view to the programmer as if their entire cluster were one giant file system.

import geotrellis.spark. import geotrellis.spark.io. import geotrellis.spark.io.hadoop._

val rootPath: Path = ... val config: Configuration = ...

/ The config argument is optional / val store: AttributeStore = HadoopAttributeStore(rootPath, config)

val reader = HadoopLayerReader(store) val writer = HadoopLayerWriter(rootPath, store)

The error is:

@.***:~/geotrellis-spark-hdfs$ sbt clean compile WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator (file:/home/sathvika/.sbt/boot/scala-2.12.19/org.scala-sbt/sbt/1.10.1/jline-terminal-3.24.1.jar) to constructor java.lang.ProcessBuilder$RedirectPipeImpl() WARNING: Please consider reporting this to the maintainers of org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release [info] welcome to sbt 1.10.1 (Oracle Corporation Java 11.0.23) [info] loading project definition from /home/sathvika/geotrellis-spark-hdfs/project [info] loading settings for project geotrellis-spark-hdfs from build.sbt ... [info] set current project to geotrellis-spark-hdfs (in build file:/home/sathvika/geotrellis-spark-hdfs/) [info] Executing in batch mode. For better performance use sbt's shell [success] Total time: 1 s, completed 03-Aug-2024, 10:47:36 PM [info] compiling 2 Scala sources to /home/sathvika/geotrellis-spark-hdfs/target/scala-2.12/classes ... [error] /home/sathvika/geotrellis-spark-hdfs/src/main/scala/try.scala:25:25: object io is not a member of package geotrellis.spark [error] import geotrellis.spark.io. [error] ^ [error] /home/sathvika/geotrellis-spark-hdfs/src/main/scala/try.scala:26:25: object io is not a member of package geotrellis.spark [error] import geotrellis.spark.io.hadoop. [error] ^ [error] two errors found [error] (Compile / compileIncremental) Compilation failed [error] Total time: 23 s, completed 03-Aug-2024, 10:47:59 PM

Could you please help me resolve the issue. Thank you for the assistance.

Message ID: @.***>

pomadchin commented 1 month ago

Hi @Cloud-1384

geotrellis-spark-hadoop is non existent. The Hadoop Attribute store now lives in geotrellis.store.hadoop._

pomadchin commented 1 month ago

Also keep in mind that all spark and hadoop dependencies are provided, meaning that there is a need to pull them into the project dependencies explicitly as they are not transitively fetched.

Cloud-1384 commented 1 month ago 
sathvika@sathvika-vm:~/geotrellis-spark-hdfs$ sbt run
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator (file:/home/sathvika/.sbt/boot/scala-2.12.19/org.scala-sbt/sbt/1.10.1/jline-terminal-3.24.1.jar) to constructor java.lang.ProcessBuilder$RedirectPipeImpl()
WARNING: Please consider reporting this to the maintainers of org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[info] welcome to sbt 1.10.1 (Oracle Corporation Java 11.0.23)
[info] loading project definition from /home/sathvika/geotrellis-spark-hdfs/project
[info] loading settings for project geotrellis-spark-hdfs from build.sbt ...
[info] set current project to geotrellis-spark-hdfs (in build file:/home/sathvika/geotrellis-spark-hdfs/)
[info] running HDFSFileChecker2 
File exists at: hdfs://localhost:9000/map.tif
[success] Total time: 10 s, completed 04-Aug-2024, 9:25:55 PM
Exception in thread "Thread-4" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/sathvika/geotrellis-spark-hdfs/target/bg-jobs/sbt_3f65c40f/target/4fe2701d/5d986717/hadoop-common-3.3.6.jar
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3098)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3045)
    at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2923)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2905)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1247)
    at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1864)
    at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1841)
    at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
    at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
    at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
Caused by: java.nio.file.NoSuchFileException: /home/sathvika/geotrellis-spark-hdfs/target/bg-jobs/sbt_3f65c40f/target/4fe2701d/5d986717/hadoop-common-3.3.6.jar
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
    at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
    at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1275)
    at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:741)
    at java.base/java.util.zip.ZipFile$CleanableResource$FinalizableResource.<init>(ZipFile.java:857)
    at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:846)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:256)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:185)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
    at java.base/sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:103)
    at java.base/sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:72)
    at java.base/sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168)
    at java.base/sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91)
    at java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
    at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:175)
    at org.apache.hadoop.conf.Configuration.parse(Configuration.java:3018)
    at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3114)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3072)
    ... 10 more

If one issue is solved the other is on its way! I am very much into geotrellis and wanted to make this work at any cost. I am trying to figure out this issue the entire day but I am unable to solve this. Please help me solve this issue? Thank you so much for your un wavering support. This means a lot to me.

pomadchin commented 1 month ago

@Cloud-1384 for that kind of usage you'd need hadoop-client & spark-core

Cloud-1384 commented 1 month ago 
Compile / mainClass := Some("HDFSFileChecker2")

version := "0.1"
scalaVersion := "2.12.18"

resolvers ++= Seq(
  "Maven Central" at "https://repo1.maven.org/maven2/",
  "Geotrellis" at "https://repo.locationtech.org/geotrellis/"
)

val geotrellisVersion = "3.7.1"
val hadoopVersion = "3.3.6"
val sparkVersion = "3.5.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.locationtech.geotrellis" %% "geotrellis-spark" % geotrellisVersion,
  "org.locationtech.geotrellis" %% "geotrellis-raster" % geotrellisVersion,
  "org.locationtech.geotrellis" %% "geotrellis-vector" % geotrellisVersion,
  "org.locationtech.geotrellis" %% "geotrellis-proj4" % geotrellisVersion,
  "org.apache.hadoop" % "hadoop-common" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-client-api" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-hdfs" % hadoopVersion
)

Is this correct?

pomadchin commented 1 month ago

The following is enough:

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.hadoop" % "hadoop-client" % hadoopVersion,
  "org.locationtech.geotrellis" %% "geotrellis-spark" % geotrellisVersion
)
Cloud-1384 commented 1 month ago

new error if I only include the above said

sathvika@sathvika-vm:~/geotrellis-spark-hdfs$ sbt run
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator (file:/home/sathvika/.sbt/boot/scala-2.12.19/org.scala-sbt/sbt/1.10.1/jline-terminal-3.24.1.jar) to constructor java.lang.ProcessBuilder$RedirectPipeImpl()
WARNING: Please consider reporting this to the maintainers of org.jline.terminal.impl.exec.ExecTerminalProvider$ReflectionRedirectPipeCreator
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[info] welcome to sbt 1.10.1 (Oracle Corporation Java 11.0.23)
[info] loading project definition from /home/sathvika/geotrellis-spark-hdfs/project
[info] loading settings for project geotrellis-spark-hdfs from build.sbt ...
[info] set current project to geotrellis-spark-hdfs (in build file:/home/sathvika/geotrellis-spark-hdfs/)
[info] running HDFSFileChecker2 
File exists at: hdfs://localhost:9000/map.tif
[success] Total time: 10 s, completed 04-Aug-2024, 9:53:32 PM
Exception in thread "Thread-5" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/sathvika/geotrellis-spark-hdfs/target/bg-jobs/sbt_38102685/target/5dca9fb7/359669fc/hadoop-client-api-3.3.4.jar
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3089)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3036)
    at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2914)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2896)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1246)
    at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1863)
    at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1840)
    at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
    at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
    at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
Caused by: java.nio.file.NoSuchFileException: /home/sathvika/geotrellis-spark-hdfs/target/bg-jobs/sbt_38102685/target/5dca9fb7/359669fc/hadoop-client-api-3.3.4.jar
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
    at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
    at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1275)
    at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:741)
    at java.base/java.util.zip.ZipFile$CleanableResource$FinalizableResource.<init>(ZipFile.java:857)
    at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:846)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:256)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:185)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
    at java.base/sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:103)
    at java.base/sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:72)
    at java.base/sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168)
    at java.base/sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91)
    at java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:132)
    at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:175)
    at org.apache.hadoop.conf.Configuration.parse(Configuration.java:3009)
    at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3105)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3063)
    ... 10 more
sathvika@sathvika-vm:~/geotrellis-spark-hdfs$

Can you please give me a code snippet through which I can fetch the data from HDFS and visualize ona web page? I will modify then according to my needs. I am trying from 1 week. I am facing new issues everyday and I guess the official documentation is also not updated so I am facing the issue.

pomadchin commented 1 month ago

@Cloud-1384 ofc, check this out: https://gist.github.com/pomadchin/63ba75dd659dc03cc2502400e361e7d8

Cloud-1384 commented 1 month ago

Thank you so much for the help :) The above code worked for me but by adding a small line of code in the build.sbt file i.e., fork := true

Any github repo for visualizing the map.tif files? I want map to be displayed on the web page using Geotrellis and SBT.

pomadchin commented 1 month ago

@Cloud-1384 have u checked https://github.com/geotrellis/geotrellis-server/blob/main/example/src/main/scala/geotrellis/server/example/ndvi/NdviService.scala ?