RECORDED_PATHS not found, failure during build

On occassion, running a build of our APK will give the following error:

[2017-06-01 16:27:43.926][error][command:null][tid:1716][com.facebook.buck.cli.Main] Uncaught exception at top level                                                                                                                 [80/1991]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Building //PATH/TO/APP#compile_r_dot_java failed. Caused by RuntimeException: Failed to read /PATH/TO/APP/.instrumentation_debug_apk#compile_r_dot_java/metadata/RECORDED_PATHS                                                            
        at com.facebook.buck.command.Build.executeAndPrintFailuresToEventBus(Build.java:382)                                                                                                                                                  
        at com.facebook.buck.cli.BuildCommand.executeBuild(BuildCommand.java:921)                                                                                                                                                             
        at com.facebook.buck.cli.BuildCommand.executeLocalBuild(BuildCommand.java:853)                                                                                                                                                        
        at com.facebook.buck.cli.BuildCommand.executeBuildAndProcessResult(BuildCommand.java:509)                                                                                                                                             
        at com.facebook.buck.cli.BuildCommand.run(BuildCommand.java:415)                                                                                                                                                                      
        at com.facebook.buck.cli.BuildCommand.runWithoutHelp(BuildCommand.java:381)                                                                                                                                                           
        at com.facebook.buck.cli.AbstractCommand.run(AbstractCommand.java:222)                                                                                                                                                                
        at com.facebook.buck.cli.AbstractContainerCommand.run(AbstractContainerCommand.java:66)                                                                                                                                               
        at com.facebook.buck.cli.BuckCommand.run(BuckCommand.java:80)                                                                                                                                                                         
        at com.facebook.buck.cli.Main.runMainWithExitCode(Main.java:900)                                                                                                                                                                      
        at com.facebook.buck.cli.Main.runMainThenExit(Main.java:329)                                                                                                                                                                          
        at com.facebook.buck.cli.Main.nailMain(Main.java:1674)                                                                                                                                                                                
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                                                        
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                                                      
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                                              
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                                                                   
        at com.martiansoftware.nailgun.NGSession.run(NGSession.java:338)                                                                                                                                                                      
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Building //PATH/TO/APP#compile_r_dot_java failed. Caused by RuntimeException: Failed to read /PATH/TO/APP/.instrumentation_debug_apk#compile_r_dot_java/metadata/RECORDED_PATHS                                                                             
        at com.facebook.buck.command.Build.executeBuild(Build.java:278)                                                                                                                                                                       
        at com.facebook.buck.command.Build.executeAndPrintFailuresToEventBus(Build.java:331)                                                                                                                                                  
        ... 16 more                                                                                                                                                                                                                           
Caused by: java.lang.RuntimeException: Building //PATH/TO/APP#compile_r_dot_java failed. Caused by RuntimeException: Failed to read /PATH/TO/APP/.instrumentation_debug_apk#compile_r_dot_java/metadata/RECORDED_PATHS                                                                                                                      
        at com.facebook.buck.rules.CachingBuildEngine.maybeAttachBuildRuleNameToException(CachingBuildEngine.java:1175)                                                                                                                       
        at com.facebook.buck.rules.CachingBuildEngine.access$12(CachingBuildEngine.java:1159)                                                                                                                                                 
        at com.facebook.buck.rules.CachingBuildEngine$2.onFailure(CachingBuildEngine.java:1118)
        at com.facebook.buck.util.concurrent.MoreFutures$1.onFailure(MoreFutures.java:138)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:137)
        at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988)
        at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$2(WeightedListeningExecutorService.java:104)
        at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$0(WeightedListeningExecutorService.java:78)
        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130)
        at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: Failed to read /PATH/TO/APP/.instrumentation_debug_apk#compile_r_dot_java/metadata/RECORDED_PATHS
        at com.facebook.buck.rules.DefaultOnDiskBuildInfo.getValuesOrThrow(DefaultOnDiskBuildInfo.java:113)
        at com.facebook.buck.rules.CachingBuildEngine.lambda$11(CachingBuildEngine.java:729)
        at com.facebook.buck.rules.CachingBuildEngine.lambda$22(CachingBuildEngine.java:1975)
        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130)
        ... 10 more
Caused by: java.nio.file.NoSuchFileException: /PATH/TO/APP/.instrumentation_debug_apk#compile_r_dot_java/metadata/RECORDED_PATHS
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)

Running buck clean and rebuilding will cause it to go away, for a little while. However, after a few build runs it often comes back up again, requiring another clean.

This is happening on buck version 744d4bb1ed004b48a9fc9a200090997bda3ca7bb

@bertmaher We have sqlite as the build metadata store, is recorded_paths something that lives outside it

cc @dreiss

RECORDED_PATHS is still stored in the filesystem, not the DB. I fixed a bug a few weeks ago (618ad57c624cec4949341f1e06f49f81983b64b6) where we'd sometimes remove metadata from the filesystem and not the DB, but that particular case shouldn't still be happening.

Does this happen to occur after a build is interrupted (Ctrl-C or something like that)? It's possible that if a build is killed at just the wrong time that the filesystem has been cleared, and not the DB. Just in case that's what's going on, I'll reorder things so we always clean the DB first, so we'll never go looking for a nonexistent RECORDED_PATHS.

Btw- switching sqlite off should make this go away, if it's a major blocker now. #sadpanda.

Thanks for the info @bertmaher . We will turn sqlite off till this is fixed

Cool. If it doesn't go away with sqlite off, let me know that too. It's entirely possible this is an unrelated bug.

Does this happen to occur after a build is interrupted (Ctrl-C or something like that)? It's possible that if a build is killed at just the wrong time that the filesystem has been cleared, and not the DB. Just in case that's what's going on, I'll reorder things so we always clean the DB first, so we'll never go looking for a nonexistent RECORDED_PATHS.

No, the build wasn't interrupted before. I've gone as far as to to a complete git clean -dfx, recompile, and the issue can still occur.

I am getting a same error here. Is it fixed or there is some workaround? How to turn sqlite off?

My error message goes like this:

[-] PROCESSING BUCK FILES...FINISHED 17.7s [100%] 🐳 New buck daemon [-] DOWNLOADING... (1.49 MB/S AVG, TOTAL: 2.76 MB, 348 Artifacts) [-] BUILDING...FINISHED 51.3s [100%] (528/3203 JOBS, 7 UPDATED, 6 [0.2%] CACHE MISS) Details: http://localhost:53844/trace/83b3a004-038f-41b6-bdc4-65687a655293 [2017-06-25 21:18:19.173][error][command:null][tid:30][com.facebook.buck.cli.Main] Uncaught exception at top level java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Building rule [//proxygen/facebook/httpclient:http_client_utils#compile-pic-EventBaseFuncWithTag.cpp.o9287e4c1,gcc-4.9-glibc-2.20,v9110ab1] failed. Caused by [ZipException]: archive is not a ZIP archive at com.facebook.buck.command.Build.executeAndPrintFailuresToEventBus(Build.java:382) at com.facebook.buck.cli.BuildCommand.executeBuild(BuildCommand.java:924) at com.facebook.buck.cli.BuildCommand.executeLocalBuild(BuildCommand.java:855) at com.facebook.buck.cli.BuildCommand.executeBuildAndProcessResult(BuildCommand.java:509) at com.facebook.buck.cli.BuildCommand.run(BuildCommand.java:415) at com.facebook.buck.cli.BuildCommand.runWithoutHelp(BuildCommand.java:381) at com.facebook.buck.cli.AbstractCommand.run(AbstractCommand.java:222) at com.facebook.buck.cli.AbstractContainerCommand.run(AbstractContainerCommand.java:66) at com.facebook.buck.cli.BuckCommand.run(BuckCommand.java:80) at com.facebook.buck.cli.Main.runMainWithExitCode(Main.java:968) at com.facebook.buck.cli.Main.runMainThenExit(Main.java:351) at com.facebook.buck.cli.Main.nailMain(Main.java:1856) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.martiansoftware.nailgun.NGSession.run(NGSession.java:338) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Building rule [//proxygen/facebook/httpclient:http_client_utils#compile-pic-EventBaseFuncWithTag.cpp.o9287e4c1,gcc-4.9-glibc-2.20,v9110ab1] failed. Caused by [ZipException]: archive is not a ZIP archive at com.facebook.buck.command.Build.executeBuild(Build.java:278) at com.facebook.buck.command.Build.executeAndPrintFailuresToEventBus(Build.java:331) ... 16 more Caused by: java.lang.RuntimeException: Building rule [//proxygen/facebook/httpclient:http_client_utils#compile-pic-EventBaseFuncWithTag.cpp.o9287e4c1,gcc-4.9-glibc-2.20,v9110ab1] failed. Caused by [ZipException]: archive is not a ZIP archive at com.facebook.buck.rules.CachingBuildEngine.maybeAttachBuildRuleNameToException(CachingBuildEngine.java:1190) at com.facebook.buck.rules.CachingBuildEngine.access$1400(CachingBuildEngine.java:117) at com.facebook.buck.rules.CachingBuildEngine$2.onFailure(CachingBuildEngine.java:1140) at com.facebook.buck.util.concurrent.MoreFutures$1.onFailure(MoreFutures.java:138) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:112) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988) at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$submit$5(WeightedListeningExecutorService.java:104) at com.facebook.buck.util.concurrent.WeightedListeningExecutorService.lambda$submitWithSemaphore$3(WeightedListeningExecutorService.java:78) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211) at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:988) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.zip.ZipException: archive is not a ZIP archive at org.apache.commons.compress.archivers.zip.ZipFile.positionAtEndOfCentralDirectoryRecord(ZipFile.java:806) at org.apache.commons.compress.archivers.zip.ZipFile.positionAtCentralDirectory(ZipFile.java:736) at org.apache.commons.compress.archivers.zip.ZipFile.populateFromCentralDirectory(ZipFile.java:481) at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:216) at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:192) at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:153) at com.facebook.buck.zip.Unzip.extractZipFile(Unzip.java:187) at com.facebook.buck.zip.Unzip.extractZipFile(Unzip.java:239) at com.facebook.buck.rules.CachingBuildEngine.unzipArtifactFromCacheResult(CachingBuildEngine.java:1487) at com.facebook.buck.rules.CachingBuildEngine.tryToFetchArtifactFromBuildCacheAndOverlayOnTopOfProjectFilesystem(CachingBuildEngine.java:1434) at com.facebook.buck.rules.CachingBuildEngine.performRuleKeyCacheCheck(CachingBuildEngine.java:583) at com.facebook.buck.rules.CachingBuildEngine.lambda$buildOrFetchFromCache$3(CachingBuildEngine.java:475) ... 8 more

@ShangxuanWu: sqlite isn't enabled by default. If you want to double-check the setting, make sure that build.metadata_storage=filesystem in your config (and not sqlite).

In any event, I don't think that error is related to SQLite; it looks like a corrupt artifact has been stored in your cache. I've only seen that happen recently on a machine with a bad disk controller.

I'm pretty sure the attached diff will fix this issue -- if you happen to give SQLite a try again, please let me know how it goes!

We are back on sqlite for a few weeks and have seen no issues so far. Thanks!

facebook / buck

RECORDED_PATHS not found, failure during build #1363