search

marcelmay / hadoop-hdfs-fsimage-exporter

Exports Hadoop HDFS content statistics to Prometheus
Apache License 2.0
152 stars 46 forks source link

Release with hfsa 1.3.10 #193

Closed DieterDP-ng closed 4 months ago

DieterDP-ng commented 4 months ago

Hi @marcelmay

I was wondering if it's possible to release a new version of this image with a new (also to be created) version of hfsa? The current versions (hfsa 1.3.9, hadoop-hdfs-fsimage-exporter 1.4.7) don't have support for Hadoop 3.4.0 yet. And it seems HDFS has changed their FSImage format in this version:

2024-07-18 10:45:06,909 [pool-3-thread-1] ERROR de.m3y.prometheus.exporter.fsimage.FsImageUpdateHandler  - Can not load FSImage /hadoop/dfs/name/current/fsimage_0000000000001369292
java.io.IOException: Unsupported layout version -67
    at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:78)
    at de.m3y.hadoop.hdfs.hfsa.core.FsImageLoader.load(FsImageLoader.java:286)
    at de.m3y.prometheus.exporter.fsimage.FsImageUpdateHandler.loadFsImage(FsImageUpdateHandler.java:207)
    at de.m3y.prometheus.exporter.fsimage.FsImageUpdateHandler.onFsImageChange(FsImageUpdateHandler.java:185)
    at de.m3y.prometheus.exporter.fsimage.FsImageWatcher.run(FsImageWatcher.java:64)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

The master branch on hfsa has already been updated, so it should just be a matter of publishing new releases.

Thank you for creating this utility!

marcelmay commented 4 months ago

Just released https://github.com/marcelmay/hadoop-hdfs-fsimage-exporter/releases/tag/release-1.4.8 including hfsa 1.3.10