Update to latest Heritrix

machawk1 commented 5 years ago

New releases beyond the 3.4.0 version that was minted by the National Library of Iceland are being pushed to https://github.com/internetarchive/heritrix3/releases. It would be good to get a newer version in WAIL beyond 3.2.0.

EDIT: Newer releases are important with regard to crawl quality and other tickets. The barrier has been the bundled Java. See comments below for mitigation plans relative to development vs. release.

EDIT 20220826: Use the heritrix-2022 branch for a newer JDK and Heritrix. OpenWayback cannot use the newer JDK, it seems, but the newer Heritrix appears to run well there.

NOTE: Heritrix releases are now available at Maven Central instead of GitHub.

[x] Get OpenWayback to work with JDK that runs newer Heritrix

machawk1 commented 5 years ago

Might need some UI adjustments for the long version number.

machawk1 commented 5 years ago

Using 3.4.0-X prevents the status update in Advanced>Heritrix

machawk1 commented 5 years ago

Initial integration in issue-345 branch.

machawk1 commented 5 years ago

5cf1067 writes WARCs from the new Heritrix but for some reason they are not indexed and accessible with the previously configured OpenWayback.

machawk1 commented 5 years ago

https://netpreserveblog.wordpress.com/2019/02/19/a-new-release-of-heritrix-3/

machawk1 commented 5 years ago

Completed more integration steps in issue-345 branch but new WARCs are still not indexed.

...despite the indexing scripts running periodically. The resulting CDX is still blank.

machawk1 commented 5 years ago

It looks like a heritrixPath + 'jobs/' directory does not exist by default. Unsure where Heritrix 3.4 is saving job information by default, but this may be the source of the issue.

machawk1 commented 5 years ago

Adding the jobs/ directory allows it to be populated when running a crawl with Heritrix 3.4.0.

...but index.cdx remains at 0 bytes.

machawk1 commented 5 years ago

It seems like the WAIL logic is tied to .warc files and does not index .warc.gz via the cdx-indexer. This should be a straightforward programmatic fix.

machawk1 commented 5 years ago

There still appears to be an issue with the CDXJ merging procedure in 1df85bcbadc67091c4a1d784d098aef71cfc7b92. The new CDXZ.GZ file is created but reset to 0 when merging with the existing index.cdx.

machawk1 commented 5 years ago

allCDXesPath = config.wailPath + "/archiveIndexes/*.cdx" is probably the culprit.

machawk1 commented 5 years ago

The current master 7027b9b with OpenWayback 2.4.0 seems to work fine with regard to indexing, so maybe let's hold off of integrating the latest Heritrix just yet until we figure out the delta in 7027b9b and 3020f99 that is preventing WARC responses from being replayable.

An interesting aside: despite the prefix for WARCs in the config in 3020f99 being MAT-, they still seem to be WEB- whereas in 7027b9b they are still MAT-.

machawk1 commented 4 years ago

Tried this again by pulling the latest release of Heritrix into the latest WAIL master (distrib package at https://github.com/internetarchive/heritrix3/releases/tag/3.4.0-20200518), started a crawl from the WAIL UI, and Heritrix never started.

WAIL CLI (run with Python) reported:

sh /Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/bin/heritrix -a lorem:ipsum
-n Thu May 28 16:47:23 EDT 2020 Heritrix starting (pid 5109)

ERROR: JVM terminated without running Heritrix.
This could be due to invalid JAVA_OPTS or JMX_PORT, etc.
See heritrix_out.log for more details.
Here are its last three lines:

    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

heritrix_out.log reported an issue:

Thu May 28 16:46:08 EDT 2020 Starting heritrix
Darwin Katja 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
JAVA_OPTS= -Xmx256m
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1392
virtual memory          (kbytes, -v) unlimited
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/archive/crawler/Heritrix : Unsupported major.minor version 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

Asked about Java version on the #heritrix channel of the IIPC Slack.

machawk1 commented 4 years ago

@ldko noted on the IIPC Slack #heritrix channel that Heritrix dropped support for Java 7 (August 2019) and to try using Java v8-11.

[ ] Bundle Java 11 with WAIL: done in issue-345-java11 branch, needs to be merged with issue-345 branch

machawk1 commented 4 years ago

In Java 11 there are some larger files in the JDK like Contents/Home/lib/modules (137.4MB) and Contents/Home/lib/src.zip (57.5 MB) that don't play well with git. Removing these for testing so the branches can be merged.

machawk1 commented 4 years ago

The issue-345-java11 branch was never pushed to GitHub. Files too big and when removed, segfault on Heritrix launch.

Added the full jdk back into the WAIL source locally without pushing to GitHub and issues still occur when trying to launch Heritrix via what is essentially the current WAIL master branch with paths adjusted for new Heritrix and JDK version.

Thu May 28 18:28:10 EDT 2020 Starting heritrix
Darwin Katja 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff7378de52, pid=29625, tid=10243
#
# JRE version:  (11.0.2+9) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_platform.dylib+0xe52]  _platform_strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/hs_err_pid29625.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
JAVA_OPTS= -Xmx256m
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1392
virtual memory          (kbytes, -v) unlimited
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff7378de52, pid=29626, tid=10499
#
# JRE version:  (11.0.2+9) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_platform.dylib+0xe52]  _platform_strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/hs_err_pid29626.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff7378de52, pid=29626, tid=10499
#
# JRE version:  (11.0.2+9) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_platform.dylib+0xe52]  _platform_strlen+0x12
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Dheritrix.home=/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518 -Djava.protocol.handler.pkgs=org.archive.net -Dheritrix.out=/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/heritrix_out.log -Xmx256m org.archive.crawler.Heritrix -a lorem:ipsum

Host: MacBookPro11,1 x86_64 2400 MHz, 4 cores, 8G, Darwin 19.4.0
Time: Thu May 28 18:28:10 2020 EDT elapsed time: 0 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007f831700b800):  JavaThread "Unknown thread" [_thread_in_vm, id=10499, stack(0x000070000f2ca000,0x000070000f3ca000)]

Stack: [0x000070000f2ca000,0x000070000f3ca000],  sp=0x000070000f3c9b70,  free space=1022k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_platform.dylib+0xe52]  _platform_strlen+0x12
V  [libjvm.dylib+0x1eb94b]  _ZN11ClassLoader22setup_boot_search_pathEPKc+0x1d
V  [libjvm.dylib+0x1ee3dc]  _ZN11ClassLoader10initializeEv+0x5d8
V  [libjvm.dylib+0x3813e5]  _Z12init_globalsv+0x2d
V  [libjvm.dylib+0x6f0d19]  _ZN7Threads9create_vmEP14JavaVMInitArgsPb+0x26d
V  [libjvm.dylib+0x3ee954]  JNI_CreateJavaVM+0x60
C  [java+0x3e58]  JavaMain+0x10b
C  [libsystem_pthread.dylib+0x6109]  _pthread_start+0x94
C  [libsystem_pthread.dylib+0x1b8b]  thread_start+0xf

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Registers:
RAX=0x0000000105b461c2, RBX=0x0000000105b49240, RCX=0x0000000000000000, RDX=0x0000000000000000
RSP=0x000070000f3c9b70, RBP=0x000070000f3c9b70, RSI=0x0000000000000000, RDI=0x0000000000000000
R8 =0x0000000000000000, R9 =0x0000000000000000, R10=0x0000000000000000, R11=0x0000000000000000
R12=0x00007f831700b800, R13=0x000070000f3c9f80, R14=0x00000001057a8b9e, R15=0x000070000f3c9ef7
RIP=0x00007fff7378de52, EFLAGS=0x0000000000010246, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x000070000f3c9b70)
0x000070000f3c9b70:   000070000f3c9c90 00000001053eb94b
0x000070000f3c9b80:   00007f8315c059f0 0000000000000040
0x000070000f3c9b90:   00007f8315e0c5c0 0000000000000040
0x000070000f3c9ba0:   000070000f3c9bd0 0000000105821684
0x000070000f3c9bb0:   0000000000000038 0000000000000040
0x000070000f3c9bc0:   00007f8315d0ff70 0000000000000002
0x000070000f3c9bd0:   000070000f3c9c30 000000010581fc54
0x000070000f3c9be0:   000070000f3c9c10 00000001057cb645
0x000070000f3c9bf0:   00007f8315c059f0 00007f8315d0ff70
0x000070000f3c9c00:   00007f8315c05e10 00007f8315e0c640
0x000070000f3c9c10:   000070000f3c9c50 00000001058202d6
0x000070000f3c9c20:   00007f8315d0ff70 00007f8315d0ff70
0x000070000f3c9c30:   0000000000000004 0000000105984164
0x000070000f3c9c40:   000000000000000b 0000000000000000
0x000070000f3c9c50:   6903b3351c140043 0000000000000000
0x000070000f3c9c60:   00000001057a8b9e 0000000105b49240
0x000070000f3c9c70:   00007f831700b800 000070000f3c9f80
0x000070000f3c9c80:   00000001057a8b9e 000070000f3c9ef7
0x000070000f3c9c90:   000070000f3c9cc0 00000001053ee3dc
0x000070000f3c9ca0:   00007f831700b800 00007f831700b800
0x000070000f3c9cb0:   0000000105b49240 00000001057a8b9e
0x000070000f3c9cc0:   000070000f3c9d10 00000001055813e5
0x000070000f3c9cd0:   00007f831700b800 00007f8315c057d0
0x000070000f3c9ce0:   00007f8315c05ed0 00007f8315c05ee0
0x000070000f3c9cf0:   00007f8315c05fb8 00000000000000d8
0x000070000f3c9d00:   00007f8315c05b10 0000000105b49240
0x000070000f3c9d10:   000070000f3c9ea0 00000001058f0d19
0x000070000f3c9d20:   0000000000000000 0000000000000000
0x000070000f3c9d30:   0000000000000000 0000000000000000
0x000070000f3c9d40:   0000000000000000 0000000000000000
0x000070000f3c9d50:   0000000000000000 0000000000000000
0x000070000f3c9d60:   0000000000000100 0000000000000000 

Instructions: (pc=0x00007fff7378de52)
0x00007fff7378de32:   90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48
0x00007fff7378de42:   89 e5 48 89 f9 48 89 fa 48 83 e7 f0 66 0f ef c0
0x00007fff7378de52:   66 0f 74 07 66 0f d7 f0 48 83 e1 0f 48 83 c8 ff
0x00007fff7378de62:   48 d3 e0 21 c6 74 17 0f bc c6 48 29 d7 48 01 f8 

---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00007f8315c05070, length=0, elements={
}

Java Threads: ( => current thread )

Other Threads:

[error occurred during error reporting (printing all threads), id 0xb, SIGSEGV (0xb) at pc=0x00000001058f2036]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

[error occurred during error reporting (printing compressed oops mode), id 0xb, SIGSEGV (0xb) at pc=0x00000001059061d2]

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events

Dynamic libraries:
0x00007fff38732000  /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
0x00007fff45f0f000  /System/Library/Frameworks/Security.framework/Versions/A/Security
0x00007fff3769c000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
0x00007fff72ab9000  /usr/lib/libz.1.dylib
0x00007fff7057d000  /usr/lib/libSystem.B.dylib
0x00007fff395cd000  /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x00007fff3bc83000  /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
0x00007fff723dd000  /usr/lib/libobjc.A.dylib
0x00007fff3688d000  /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
0x00007fff390b6000  /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
0x00007fff6a59e000  /System/Library/PrivateFrameworks/UIFoundation.framework/Versions/A/UIFoundation
0x00007fff6564c000  /System/Library/PrivateFrameworks/RemoteViewServices.framework/Versions/A/RemoteViewServices
0x00007fff6d0f2000  /System/Library/PrivateFrameworks/XCTTargetBootstrap.framework/Versions/A/XCTTargetBootstrap
0x00007fff394a1000  /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
0x00007fff3eb3c000  /System/Library/Frameworks/Metal.framework/Versions/A/Metal
0x00007fff536a5000  /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
0x00007fff716ba000  /usr/lib/liblangid.dylib
0x00007fff5293e000  /System/Library/PrivateFrameworks/CoreSVG.framework/Versions/A/CoreSVG
0x00007fff687a2000  /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
0x00007fff39a4e000  /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
0x00007fff35261000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x00007fff729aa000  /usr/lib/libxml2.2.dylib
0x00007fff5e5fe000  /System/Library/PrivateFrameworks/IconServices.framework/Versions/A/IconServices
0x00007fff3c462000  /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
0x00007fff700a7000  /usr/lib/libDiagnosticMessagesClient.dylib
0x00007fff534ff000  /System/Library/PrivateFrameworks/DFRFoundation.framework/Versions/A/DFRFoundation
0x00007fff71449000  /usr/lib/libicucore.A.dylib
0x00007fff379ce000  /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
0x00007fff37ab1000  /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
0x00007fff7077f000  /usr/lib/libauto.dylib
0x00007fff535a1000  /System/Library/PrivateFrameworks/DataDetectorsCore.framework/Versions/A/DataDetectorsCore
0x00007fff38253000  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
0x00007fff4510c000  /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
0x00007fff38592000  /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
0x00007fff53016000  /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
0x00007fff38b21000  /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
0x00007fff3b948000  /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
0x00007fff6064b000  /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
0x00007fff71310000  /usr/lib/libenergytrace.dylib
0x00007fff3c3bc000  /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x00007fff3a9b5000  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
0x00007fff627e2000  /System/Library/PrivateFrameworks/PerformanceAnalysis.framework/Versions/A/PerformanceAnalysis
0x00007fff4413f000  /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
0x00007fff38740000  /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
0x00007fff3a0d0000  /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
0x00007fff3b427000  /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
0x00007fff3c4f2000  /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
0x00007fff70863000  /usr/lib/libc++.1.dylib
0x00007fff708df000  /usr/lib/libcompression.dylib
0x00007fff703eb000  /usr/lib/libMobileGestalt.dylib
0x00007fff6a317000  /System/Library/PrivateFrameworks/TextureIO.framework/Versions/A/TextureIO
0x00007fff706e2000  /usr/lib/libate.dylib
0x00007fff5e807000  /System/Library/PrivateFrameworks/InternationalSupport.framework/Versions/A/InternationalSupport
0x00007fff73379000  /usr/lib/system/libcache.dylib
0x00007fff7337f000  /usr/lib/system/libcommonCrypto.dylib
0x00007fff7338b000  /usr/lib/system/libcompiler_rt.dylib
0x00007fff73393000  /usr/lib/system/libcopyfile.dylib
0x00007fff7339d000  /usr/lib/system/libcorecrypto.dylib
0x00007fff7353c000  /usr/lib/system/libdispatch.dylib
0x00007fff7357d000  /usr/lib/system/libdyld.dylib
0x00007fff735b4000  /usr/lib/system/libkeymgr.dylib
0x00007fff735c2000  /usr/lib/system/liblaunch.dylib
0x00007fff735c3000  /usr/lib/system/libmacho.dylib
0x00007fff735c9000  /usr/lib/system/libquarantine.dylib
0x00007fff735cc000  /usr/lib/system/libremovefile.dylib
0x00007fff735ce000  /usr/lib/system/libsystem_asl.dylib
0x00007fff735e6000  /usr/lib/system/libsystem_blocks.dylib
0x00007fff735e7000  /usr/lib/system/libsystem_c.dylib
0x00007fff7366f000  /usr/lib/system/libsystem_configuration.dylib
0x00007fff73673000  /usr/lib/system/libsystem_coreservices.dylib
0x00007fff73677000  /usr/lib/system/libsystem_darwin.dylib
0x00007fff73680000  /usr/lib/system/libsystem_dnssd.dylib
0x00007fff73688000  /usr/lib/system/libsystem_featureflags.dylib
0x00007fff7368a000  /usr/lib/system/libsystem_info.dylib
0x00007fff73705000  /usr/lib/system/libsystem_m.dylib
0x00007fff7374d000  /usr/lib/system/libsystem_malloc.dylib
0x00007fff73775000  /usr/lib/system/libsystem_networkextension.dylib
0x00007fff73783000  /usr/lib/system/libsystem_notify.dylib
0x00007fff737a1000  /usr/lib/system/libsystem_sandbox.dylib
0x00007fff737a6000  /usr/lib/system/libsystem_secinit.dylib
0x00007fff736d8000  /usr/lib/system/libsystem_kernel.dylib
0x00007fff7378d000  /usr/lib/system/libsystem_platform.dylib
0x00007fff73796000  /usr/lib/system/libsystem_pthread.dylib
0x00007fff737a9000  /usr/lib/system/libsystem_symptoms.dylib
0x00007fff737b1000  /usr/lib/system/libsystem_trace.dylib
0x00007fff737c9000  /usr/lib/system/libunwind.dylib
0x00007fff737cf000  /usr/lib/system/libxpc.dylib
0x00007fff708b6000  /usr/lib/libc++abi.dylib
0x00007fff716bc000  /usr/lib/liblzma.5.dylib
0x00007fff71337000  /usr/lib/libfakelink.dylib
0x00007fff7066f000  /usr/lib/libarchive.2.dylib
0x00007fff46fbf000  /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
0x00007fff6ffda000  /usr/lib/libCRFSuite.dylib
0x00007fff37e44000  /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
0x00007fff70845000  /usr/lib/libbsm.0.dylib
0x00007fff735b5000  /usr/lib/system/libkxld.dylib
0x00007fff4ced4000  /System/Library/PrivateFrameworks/AppleFSCompression.framework/Versions/A/AppleFSCompression
0x00007fff70bd0000  /usr/lib/libcoretls.dylib
0x00007fff70be7000  /usr/lib/libcoretls_cfhelpers.dylib
0x00007fff72423000  /usr/lib/libpam.2.dylib
0x00007fff72558000  /usr/lib/libsqlite3.dylib
0x00007fff72997000  /usr/lib/libxar.1.dylib
0x00007fff70856000  /usr/lib/libbz2.1.0.dylib
0x00007fff71358000  /usr/lib/libiconv.2.dylib
0x00007fff708cc000  /usr/lib/libcharset.1.dylib
0x00007fff71ec2000  /usr/lib/libnetwork.dylib
0x00007fff7242a000  /usr/lib/libpcap.A.dylib
0x00007fff70624000  /usr/lib/libapple_nghttp2.dylib
0x00007fff3ad6c000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
0x00007fff3aa3c000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x00007fff3afaf000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x00007fff3b048000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x00007fff3b076000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
0x00007fff3a9b6000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
0x00007fff3ad75000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
0x00007fff3ad1e000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
0x00007fff3b0de000  /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
0x00007fff3ff5f000  /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
0x00007fff60b7a000  /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
0x00007fff6d4bf000  /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
0x00007fff69de4000  /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
0x00007fff51ab1000  /System/Library/PrivateFrameworks/CoreNLP.framework/Versions/A/CoreNLP
0x00007fff60079000  /System/Library/PrivateFrameworks/MetadataUtilities.framework/Versions/A/MetadataUtilities
0x00007fff71794000  /usr/lib/libmecabra.dylib
0x00007fff716ec000  /usr/lib/libmecab.dylib
0x00007fff71348000  /usr/lib/libgermantok.dylib
0x00007fff7060b000  /usr/lib/libThaiTokenizer.dylib
0x00007fff70011000  /usr/lib/libChineseTokenizer.dylib
0x00007fff35279000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
0x00007fff366e3000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
0x00007fff36620000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
0x00007fff36448000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
0x00007fff358d0000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
0x00007fff3600c000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
0x00007fff363a8000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
0x00007fff36435000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
0x00007fff363be000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
0x00007fff35b38000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
0x00007fff363c4000  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
0x00007fff5ebef000  /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
0x00007fff51461000  /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
0x00007fff5ed0d000  /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
0x00007fff5ecbe000  /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
0x00007fff708cd000  /usr/lib/libcmph.dylib
0x00007fff42b4d000  /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
0x00007fff42b6a000  /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
0x00007fff4af8a000  /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
0x00007fff46259000  /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
0x00007fff72993000  /usr/lib/libutil.dylib
0x00007fff52996000  /System/Library/PrivateFrameworks/CoreServicesStore.framework/Versions/A/CoreServicesStore
0x00007fff46310000  /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
0x00007fff4e6a2000  /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
0x00007fff72a90000  /usr/lib/libxslt.1.dylib
0x00007fff4d546000  /System/Library/PrivateFrameworks/AppleSystemInfo.framework/Versions/A/AppleSystemInfo
0x00007fff3c70d000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
0x00007fff3c9c9000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
0x00007fff3c9ac000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
0x00007fff3c64f000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
0x00007fff3c653000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
0x00007fff3c9c7000  /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
0x00007fff71311000  /usr/lib/libexpat.1.dylib
0x00007fff4d02f000  /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
0x00007fff55c71000  /System/Library/PrivateFrameworks/FontServices.framework/libFontParser.dylib
0x00007fff6c4c5000  /System/Library/PrivateFrameworks/WatchdogClient.framework/Versions/A/WatchdogClient
0x00007fff5e24b000  /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
0x00007fff3eed7000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
0x00007fff5a9e4000  /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
0x00007fff5e260000  /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
0x00007fff53512000  /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
0x00007fff434de000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
0x00007fff3ec23000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSCore.framework/Versions/A/MPSCore
0x00007fff3ec61000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSImage.framework/Versions/A/MPSImage
0x00007fff3ed28000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
0x00007fff3ecec000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
0x00007fff3ee87000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSRayIntersector.framework/Versions/A/MPSRayIntersector
0x00007fff3ed12000  /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNDArray.framework/Versions/A/MPSNDArray
0x00007fff600c6000  /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
0x00007fff4c15c000  /System/Library/PrivateFrameworks/AggregateDictionary.framework/Versions/A/AggregateDictionary
0x00007fff50ebb000  /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/CoreAnalytics
0x00007fff4d465000  /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
0x00007fff702cd000  /usr/lib/libIOReport.dylib
0x00007fff3b5df000  /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
0x00007fff5be5e000  /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
0x00007fff551c5000  /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
0x00007fff42af4000  /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
0x00007fff700ef000  /usr/lib/libFosl_dynamic.dylib
0x00007fff6158f000  /System/Library/PrivateFrameworks/OTSVG.framework/Versions/A/OTSVG
0x00007fff377a1000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
0x00007fff55e42000  /System/Library/PrivateFrameworks/FontServices.framework/libhvf.dylib
0x00007fff434e9000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
0x00007fff436c7000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
0x00007fff434f2000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
0x00007fff434fd000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
0x00007fff434db000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
0x00007fff434e4000  /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
0x00007fff71d63000  /usr/lib/libncurses.5.4.dylib
0x00007fff3769d000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
0x00007fff3786a000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
0x00007fff37908000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
0x00007fff37960000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
0x00007fff3796f000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
0x00007fff379b5000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
0x00007fff379c0000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
0x00007fff3783a000  /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATSUI.framework/Versions/A/ATSUI
0x00007fff711a5000  /usr/lib/libcups.2.dylib
0x00007fff3df74000  /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
0x00007fff3c0b5000  /System/Library/Frameworks/GSS.framework/Versions/A/GSS
0x00007fff724e4000  /usr/lib/libresolv.9.dylib
0x00007fff5c00c000  /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
0x00007fff3df87000  /System/Library/Frameworks/Kerberos.framework/Versions/A/Libraries/libHeimdalProxy.dylib
0x00007fff7134e000  /usr/lib/libheimdal-asn1.dylib
0x00007fff50354000  /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
0x00007fff4d5e3000  /System/Library/PrivateFrameworks/AssertionServices.framework/Versions/A/AssertionServices
0x00007fff4e185000  /System/Library/PrivateFrameworks/AudioToolboxCore.framework/Versions/A/AudioToolboxCore
0x00007fff6d16f000  /System/Library/PrivateFrameworks/caulk.framework/Versions/A/caulk
0x00007fff4e74e000  /System/Library/PrivateFrameworks/BaseBoard.framework/Versions/A/BaseBoard
0x00007fff657da000  /System/Library/PrivateFrameworks/RunningBoardServices.framework/Versions/A/RunningBoardServices
0x00007fff627ee000  /System/Library/PrivateFrameworks/PersistentConnection.framework/Versions/A/PersistentConnection
0x00007fff651d4000  /System/Library/PrivateFrameworks/ProtocolBuffer.framework/Versions/A/ProtocolBuffer
0x00007fff50378000  /System/Library/PrivateFrameworks/CommonUtilities.framework/Versions/A/CommonUtilities
0x00007fff4e8dc000  /System/Library/PrivateFrameworks/Bom.framework/Versions/A/Bom
0x00007fff6ffa0000  /usr/lib/libAudioToolboxUtility.dylib
0x00007fff4e6ac000  /System/Library/PrivateFrameworks/Backup.framework/Versions/A/Backup
0x00007fff53433000  /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
0x00007fff67268000  /System/Library/PrivateFrameworks/Sharing.framework/Versions/A/Sharing
0x00007fff4cbf2000  /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Apple80211
0x00007fff4e3c9000  /System/Library/PrivateFrameworks/AuthKit.framework/Versions/A/AuthKit
0x00007fff53143000  /System/Library/PrivateFrameworks/CoreUtils.framework/Versions/A/CoreUtils
0x00007fff3b624000  /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
0x00007fff3c242000  /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth
0x00007fff603ca000  /System/Library/PrivateFrameworks/MobileKeyBag.framework/Versions/A/MobileKeyBag
0x00007fff51f49000  /System/Library/PrivateFrameworks/CorePhoneNumbers.framework/Versions/A/CorePhoneNumbers
0x00007fff4cfe2000  /System/Library/PrivateFrameworks/AppleIDAuthSupport.framework/Versions/A/AppleIDAuthSupport
0x00007fff3ff6c000  /System/Library/Frameworks/Network.framework/Versions/A/Network
0x00007fff5ea9b000  /System/Library/PrivateFrameworks/KeychainCircle.framework/Versions/A/KeychainCircle
0x00007fff3907d000  /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth
0x00007fff692e5000  /System/Library/PrivateFrameworks/SpeechRecognitionCore.framework/Versions/A/SpeechRecognitionCore
0x0000000105200000  /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/lib/server/libjvm.dylib
0x000000010492a000  /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/lib/libverify.dylib
0x0000000104936000  /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/lib/libjava.dylib
0x00000001049c5000  /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/lib/libzip.dylib
0x00000001049cd000  /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/lib/libjimage.dylib

VM Arguments:
jvm_args: -Dheritrix.home=/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518 -Djava.protocol.handler.pkgs=org.archive.net -Dheritrix.out=/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/heritrix_out.log -Xmx256m 
java_command: org.archive.crawler.Heritrix -a lorem:ipsum
java_class_path (initial): :/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/FastInfoset-1.2.15.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/ant-1.7.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/ant-launcher-1.7.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/aopalliance-1.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/bsh-2.0b5.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-cli-1.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-codec-1.6.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-collections-3.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-configuration-1.8.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-httpclient-3.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-io-1.4.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-lang-2.6.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-logging-1.0.4.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-math3-3.1.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-net-2.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/dnsjava-2.0.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/dsiutils-2.0.12.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/fastutil-6.5.2.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/freemarker-2.3.19.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/groovy-all-1.6.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/guava-17.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/hamcrest-core-1.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/heritrix-commons-3.4.0-20200518.jar:/Applications/WAI
Launcher Type: SUN_STANDARD

[Global flags]
     uint G1ConcRefinementThreads                  = 4                                         {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 134217728                                 {product} {ergonomic}
   size_t MaxHeapSize                              = 268435456                                 {product} {command line}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/
JRE_HOME=/Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/jdk-11.0.2.jdk/Contents/Home/
CLASSPATH=:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/FastInfoset-1.2.15.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/ant-1.7.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/ant-launcher-1.7.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/aopalliance-1.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/bsh-2.0b5.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-cli-1.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-codec-1.6.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-collections-3.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-configuration-1.8.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-httpclient-3.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-io-1.4.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-lang-2.6.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-logging-1.0.4.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-math3-3.1.1.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/commons-net-2.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/dnsjava-2.0.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/dsiutils-2.0.12.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/fastutil-6.5.2.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/freemarker-2.3.19.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/groovy-all-1.6.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/guava-17.0.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/hamcrest-core-1.3.jar:/Applications/WAIL.app/bundledApps/heritrix-3.4.0-20200518/lib/heritrix-commons-3.4.0-20200518.jar:/Applications/WAIL.app/bundledApps/heritrix-
PATH=/usr/local/opt/android-sdk/build-tools/19.1.0:/usr/local/opt/android-sdk/platform-tools:/usr/local/opt/android-sdk/tools:/usr/local/opt/gradle/bin:/usr/local/opt/maven/bin:/usr/local/opt/ant/bin:/Users/machawk1/android-platform-tools:/usr/local/npm_packages/bin/:/Users/machawk1/.npm-packages/bin:/Users/machawk1/.npm-packages/bin:/usr/local/bin:/Users/machawk1/go//bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:# homebrews should always take precedence:/usr/local/bin:# the default stack:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/usr/local/share/dotnet:/opt/X11/bin:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS:/Users/machawk1/android-sdks/platform-tools/:/Users/machawk1/gs-venv/bin
SHELL=/bin/zsh
DISPLAY=/private/tmp/com.apple.launchd.SVXBXNbeun/org.macosforge.xquartz:0

Signal Handlers:
SIGSEGV: [libjvm.dylib+0x73f4e4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_ONSTACK|SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.dylib+0x73f4e4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.dylib+0x73f4e4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.dylib+0x5f2192], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.dylib+0x5f2192], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.dylib+0x73f4e4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.dylib+0x5f2777], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART
SIGQUIT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none

---------------  S Y S T E M  ---------------

OS:uname:Darwin 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
rlimit: STACK 8192k, CORE 0k, NPROC 1392, NOFILE 10240, AS infinity, DATA infinity, FSIZE infinity
load average:2.44 2.32 2.40

CPU:total 4 (initial active 4) 

Memory: 4k page, physical 8388608k(99492k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.2+9) for bsd-amd64 JRE (11.0.2+9), built on Jan 18 2019 08:16:22 by "mach5one" with gcc 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)

END.

machawk1 commented 3 years ago

I attempted to update to the latest Heritrix and using the JDK 11 zulu16.30.19-ca-jdk16.0.1-macosx_aarch64 but again, the files are too large for GitHub. There should be a way to remotely pull in this asset at build time.

Current issue-345 branch has latest Heritrix but it requires a newer Java than 7.

machawk1 commented 3 years ago

A related aside, the indexing procedure in OpenWayback uses the filename of the WARC as part of the basis of the CDX by default. Newer versions of Heritrix compress using gzip by default (?), so .cdx.gz files are created. The indexing/coalescence procedure in WAIL is only checking for *.cdx in the globbing prior to combining and sorting.

machawk1 commented 3 years ago

A new version was released on August 3, 2021: https://github.com/internetarchive/heritrix3/releases/tag/3.4.0-20210803

machawk1 commented 3 years ago

An alternative to bundling Heritrix might be to use the submodule approach and provide a reference in WAIL to the latest Heritrix and tweak as needed for WAIL or pull the dist from the Heritrix releases and tweak if the package in the format is not needed. Preliminary experiments with newer versions of Heritrix than 3.2.0 as bundled in the most recent release indicate that the same crawl configurations in the newer versions of Heritrix (e.g., issues like #458) would be remedied with a newer version of Heritrix, so this GH issue ought to be a priority (thus pinned).

machawk1 commented 3 years ago

The submodule approach would work with Heritrix, but the issue is not the size of Heritrix but the size of the JDK that needs to be bundled for Heritrix to run. GitHub repos alone are insufficient for files the size of those contained in the JDK.

machawk1 commented 3 years ago

This is a detail of packaging for release.

[ ] Add instructions for development for pulling in a recent version of Java.
[ ] Release a version that bundled up a recent JDK despite it not residing in the repo.

machawk1 commented 2 years ago

In the heritrix-2022 branch I have added the latest Heritrix binary and a more recent JDK as required. The JDK does NOT include a "modules" file at 120+ MB, which is problematic to store in git. In lieu, the macOS build script now fetches this file at build time.

machawk1 commented 2 years ago

Attempting to replicate the current state of the heritrix-2022 branch produces an issue with architecture when run on x86_64 (Mac Intel). Original prototyping was on arm64 (Apple Silicon). To remedy this, ensure that the architecture is correct for the JDK being used.

EDIT: We only pull in the large modules file from the JDK. The rest of the JDK is in this repo.

machawk1 commented 2 years ago

After detecting architecture and serving the respective modules file, an x86_64 machine still complains about the architecture based on the java binary.

zsh: bad CPU type in executable: /Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home//bin/java

Thus, more than just the modules file is needed from the JDK release.

machawk1 commented 2 years ago

[ ] Add checksum verification for JVM download.

Also, some bash logic for detecting platform and pulling in the respective platform's modules file is present in the heritrix-2022-javafetch branch.

machawk1 commented 1 year ago

OpenWayback has issues starting w/ the alternative JDK/JRE in the heritrix-2022 branch.

% export JAVA_HOME=/Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/
% export JAVA_HOME=/Applications/WAIL.app/bundledApps/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/
% /Applications/WAIL.app/bundledApps/tomcat/bin/startup.sh

produces a log in /Applications/WAIL.app/bundledApps/tomcat/logs/catalina.out of:

-Djava.endorsed.dirs=/Applications/WAIL.app/bundledApps/tomcat/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

EDIT: https://stackoverflow.com/questions/46419496/how-do-you-replace-endorsed-directory-in-java-9 indicates that -Djava.endorsed.dirs is not supported by newer Javas, so it might be a matter of removing that invocation from the Tomcat startup.

EDIT2: This flag is used multiple times in catalina.sh

EDIT3: Around line 397 in catalina.sh, the command:

eval \"$_RUNJAVA\" \"$LOGGING_CONFIG\" $JAVA_OPTS $CATALINA_OPTS \
  -Djava.endorsed.dirs=\"$JAVA_ENDORSED_DIRS\" -classpath \"$CLASSPATH\" \
  -Dcatalina.base=\"$CATALINA_BASE\" \
  -Dcatalina.home=\"$CATALINA_HOME\" \
  -Djava.io.tmpdir=\"$CATALINA_TMPDIR\" \
  org.apache.catalina.startup.Bootstrap "$@" start \
  >> "$CATALINA_OUT" 2>&1 "&"

is the culprit, namely the java.endorsed.dirs. Modifying the command to read:

eval \"$_RUNJAVA\" \"$LOGGING_CONFIG\" $JAVA_OPTS $CATALINA_OPTS \
  -classpath \"$CLASSPATH\" \
  -Dcatalina.base=\"$CATALINA_BASE\" \
  -Dcatalina.home=\"$CATALINA_HOME\" \
  -Djava.io.tmpdir=\"$CATALINA_TMPDIR\" \
  org.apache.catalina.startup.Bootstrap "$@" start \
  >> "$CATALINA_OUT" 2>&1 "&"

...appears to start the tomcat service (it's displayed in ps output) but accessing localhost:8080 produces an error

HTTP Status 500 - Unable to compile class for JSP: type Exception report message Unable to compile class for JSP: description The server encountered an internal error that prevented it from fulfilling this request. exception org.apache.jasper.JasperException: Unable to compile class for JSP: An error occurred at line: 1 in the generated java file The type java.util.Map$Entry cannot be resolved. It is indirectly referenced from required .class files An error occurred at line: 1 in the generated java file The type java.lang.String cannot be resolved. It is indirectly referenced from required .class files An error occurred at line: 1 in the generated java file The type java.io.ObjectInputStream cannot be resolved. It is indirectly referenced from required .class files An error occurred at line: 24 in the generated java file java.lang.String cannot be resolved to a type An error occurred at line: 29 in the generated java file java.lang.String cannot be resolved to a type An error occurred at line: 30 in the generated java file String cannot be resolved to a type An error occurred at line: 55 in the generated java file The method setContentType(String) from the type ServletResponse refers to the missing type String An error occurred at line: 56 in the generated java file The method getPageContext(Servlet, ServletRequest, ServletResponse, String, boolean, int, boolean) from the type JspFactory refers to the missing type String An error occurred at line: 1 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 1: <%@ page language="java" pageEncoding="utf-8" contentType="text/html;charset=utf-8"%> 2: <%@ page import="java.util.ArrayList" %> 3: <%@ page import="org.archive.wayback.core.UIResults" %> 4: <%@ page import="org.archive.wayback.util.StringFormatter" %> An error occurred at line: 1 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 1: <%@ page language="java" pageEncoding="utf-8" contentType="text/html;charset=utf-8"%> 2: <%@ page import="java.util.ArrayList" %> 3: <%@ page import="org.archive.wayback.core.UIResults" %> 4: <%@ page import="org.archive.wayback.util.StringFormatter" %> An error occurred at line: 1 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 1: <%@ page language="java" pageEncoding="utf-8" contentType="text/html;charset=utf-8"%> 2: <%@ page import="java.util.ArrayList" %> 3: <%@ page import="org.archive.wayback.core.UIResults" %> 4: <%@ page import="org.archive.wayback.util.StringFormatter" %> An error occurred at line: 1 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 1: <%@ page language="java" pageEncoding="utf-8" contentType="text/html;charset=utf-8"%> 2: <%@ page import="java.util.ArrayList" %> 3: <%@ page import="org.archive.wayback.core.UIResults" %> 4: <%@ page import="org.archive.wayback.util.StringFormatter" %> An error occurred at line: 5 in the jsp file: /index.jsp The method include(ServletRequest, ServletResponse, String, JspWriter, boolean) from the type JspRuntimeLibrary refers to the missing type String 2: <%@ page import="java.util.ArrayList" %> 3: <%@ page import="org.archive.wayback.core.UIResults" %> 4: <%@ page import="org.archive.wayback.util.StringFormatter" %> 5: 6: <% 7: UIResults results = UIResults.getGeneric(request); 8: StringFormatter fmt = results.getWbRequest().getFormatter(); An error occurred at line: 9 in the jsp file: /index.jsp The method getAttribute(String) from the type ServletRequest refers to the missing type String 6: <% 7: UIResults results = UIResults.getGeneric(request); 8: StringFormatter fmt = results.getWbRequest().getFormatter(); 9: Object names = request.getAttribute("AccessPointNames"); 10: if(names != null) { 11: if(names instanceof ArrayList) { 12: ArrayList accessPoints = (ArrayList) names; An error occurred at line: 12 in the jsp file: /index.jsp String cannot be resolved to a type 9: Object names = request.getAttribute("AccessPointNames"); 10: if(names != null) { 11: if(names instanceof ArrayList) { 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> An error occurred at line: 12 in the jsp file: /index.jsp String cannot be resolved to a type 9: Object names = request.getAttribute("AccessPointNames"); 10: if(names != null) { 11: if(names instanceof ArrayList) { 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> An error occurred at line: 14 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 11: if(names instanceof ArrayList) { 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } An error occurred at line: 14 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 11: if(names instanceof ArrayList) { 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } An error occurred at line: 15 in the jsp file: /index.jsp The method format(String, Object...) from the type StringFormatter refers to the missing type String 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } 18: for(String accessPoint : accessPoints) { An error occurred at line: 15 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } 18: for(String accessPoint : accessPoints) { An error occurred at line: 15 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 12: ArrayList accessPoints = (ArrayList) names; 13: if(accessPoints.size() > 0) { 14: %> 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } 18: for(String accessPoint : accessPoints) { An error occurred at line: 18 in the jsp file: /index.jsp String cannot be resolved to a type 15: <%= fmt.format("UIGlobal.wrongAccesspoint") %> 16: <% 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% An error occurred at line: 19 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 16: <% 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% 22: } An error occurred at line: 19 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 16: <% 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% 22: } An error occurred at line: 20 in the jsp file: /index.jsp The method getContextPath() from the type HttpServletRequest refers to the missing type String 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% 22: } 23: } An error occurred at line: 20 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% 22: } 23: } An error occurred at line: 20 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 17: } 18: for(String accessPoint : accessPoints) { 19: %> 20: /"><%= accessPoint %>

21: <% 22: } 23: } An error occurred at line: 25 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 22: } 23: } 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

An error occurred at line: 25 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 22: } 23: } 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

An error occurred at line: 25 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 22: } 23: } 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

An error occurred at line: 27 in the jsp file: /index.jsp The method format(String, Object...) from the type StringFormatter refers to the missing type String 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

29: An error occurred at line: 27 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

29: An error occurred at line: 27 in the jsp file: /index.jsp The method write(String) from the type Writer refers to the missing type String 24: } 25: %> 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

29: An error occurred at line: 29 in the jsp file: /index.jsp The method include(ServletRequest, ServletResponse, String, JspWriter, boolean) from the type JspRuntimeLibrary refers to the missing type String 26:

27: <%= fmt.format("UIGlobal.indexPage") %> 28:

29: Stacktrace: org.apache.jasper.compiler.DefaultErrorHandler.javacError(DefaultErrorHandler.java:102) org.apache.jasper.compiler.ErrorDispatcher.javacError(ErrorDispatcher.java:331) org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:469) org.apache.jasper.compiler.Compiler.compile(Compiler.java:378) org.apache.jasper.compiler.Compiler.compile(Compiler.java:353) org.apache.jasper.compiler.Compiler.compile(Compiler.java:340) org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:646) org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:357) org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390) org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334) javax.servlet.http.HttpServlet.service(HttpServlet.java:722) org.archive.wayback.util.webapp.RequestFilter.doFilter(RequestFilter.java:157) note The full stack trace of the root cause is available in the Apache Tomcat/7.0.30 logs. Apache Tomcat/7.0.30

It might be that this newer version of Java and older version of Tomcat are not compatible. Some details:

/Applications/WAIL.app/bundledApps/tomcat/bin/bootstrap.jar:/Applications/WAIL.app/bundledApps/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/7.0.30
Server built:   Sep 2 2012 09:50:47
Server number:  7.0.30.0
OS Name:        Mac OS X
OS Version:     13.2.1
Architecture:   aarch64
JVM Version:    17.0.4.1+1
JVM Vendor:     Eclipse Adoptium

[x] Test with newer version of Tomcat and Temurin-17

machawk1 commented 1 year ago

I updated to tomcat 10 in the heritrix-2022 branch but starting Tomcat using startup.sh after setting execution permissions on catalina.sh shows a 404 in the browser. Tomcat starts fine but the OpenWayback app is not initialized.

machawk1 commented 1 year ago

The error in catalina.out:

30-Mar-2023 17:18:09.434 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/Applications/WAIL.app/bundledApps/tomcat/webapps/ROOT]
Mar 30, 2023 5:18:11 PM org.apache.catalina.core.StandardContext filterStart
SEVERE: Exception starting filter [RequestFilter]
java.lang.NoClassDefFoundError: javax/servlet/Filter
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1012)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
    at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2487)
    at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:865)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1401)
(clipped)

is quite similar to https://github.com/iipc/openwayback/issues/169.

Was this change merged into the main branch of OpenWayback?

machawk1 commented 1 year ago

Per the change in https://github.com/iipc/openwayback/pull/198 -- within WAIL's OpenWayback, the logging-config-path and config-path are still relative, i.e., /WEB-INF/classes/logging.properties and /WEB-INF/wayback.xml , respectively.

EDIT: correction, the PR has the paths lead by a preceding "/" to imply absolute. This is how it is in WAIL's OpenWayback. A complete absolute path relative to the file system does not remedy the issue.

machawk1 commented 1 month ago

Note that, as described in https://github.com/internetarchive/heritrix3/issues/237, some of the beans like BDBCookieStorage have been changed (for example) to BDBCookieStore. The crawler configs generated by Heritrix ought to match what it expects, but WAIL generates its own crawl configs, which won't match the expectations of the crawler.

machawk1 commented 1 month ago

A lot was done in the heritrix-update-oct-2024 branch. The latest Heritrix is working there. We also mitigated the large files in Git from JDK 17. An issue remains there that despite cdx-indexer generating a CDX file in the right location based on the WARCs that Heritrix created and an entry is displayed in the calendar view, on access OpenWayback says resource not available.

Both JDK 1.7 and JDK 17 are included to account for the limitations of OpenWayback and Heritrix, respectively.

machawk1 commented 1 month ago

I believe there is a race process that is first writing filename information to the index.cdx file then the CDX data is actually populated. This confuses the OpenWayback. Keep the index.cdx file open in VS Code to see it toggle between one line and many.

machawk1 commented 1 month ago

self.generate_path_index() seems to be the culprit of the undesired behavior.

machawk1 commented 1 month ago

fdd839b00e64de0bce94ae848e7a9474ce45fc67 fixes this issue. Heritrix-provided URLs are crawled and replayable! 🙌

machawk1 commented 1 month ago

Tried this on another Apple MX machine from source fdd839b00e64de0bce94ae848e7a9474ce45fc67 and upon attempting to start a job, Heritrix 3.4.0-20240909 reports:

SEVERE Failed to start bean 'bdb'; nested exception is java.lang.RuntimeException: com.sleepycat.je.DiskLimitException: (JE 7.5.11) Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited: maxDiskLimit=0 freeDiskLimit=5,368,709,120 adjustedMaxDiskLimit=0 maxDiskOverage=0 freeDiskShortage=396,890,112 diskFreeSpace=4,971,819,008 availableLogSize=-396,890,112 totalLogSize=1,266 activeLogSize=1,266 reservedLogSize=0 protectedLogSize=0 protectedLogSizeMap={} (in thread '1728049223 launchthread')

EDIT: related discussion from the Heritrix repo: https://github.com/internetarchive/heritrix3/issues/340 EDIT: Maybe it's a space available issue. Andy says BDB corrupted under 5GB and I have 4 on this machine. I'll free up some space and try again.

machawk1 / wail

Update to latest Heritrix #345