Finish Infer Integration

La0 commented 6 years ago

In order to finish @ReapOmen integration of Infer on the code review bot:

[x] Fix this issue encoutered during check-java run (from this staging task):
```
FAILURE: Build completed with 2 failures.
```

1: Task failed with an exception.

What went wrong: A problem was found with the configuration of task \':app:packageOfficialWithoutGeckoBinariesNoMinApiPhotonDebug\'.

File \'/cache/sa-unified/mobile/android/app/?/.android/debug.keystore\' specified for property \'signingConfig.storeFile\' does not exist.
Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

2: Task failed with an exception.

What went wrong: A problem was found with the configuration of task \':app:packageOfficialWithoutGeckoBinariesNoMinApiPhotonDebugAndroidTest\'.

File \'/cache/sa-unified/mobile/android/app/?/.android/debug.keystore\' specified for property \'signingConfig.storeFile\' does not exist.
Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
[x] Display Infer issues cleanly in staticanalysis/frontend details

abpostelnicu commented 6 years ago

One problem that I see here, is that the actual command from 'mach', 'check-java' doesn't pre-configure the build. At least locally this seems to solve the issue.

rbartlensky commented 6 years ago

@abpostelnicu Before running infer, the bot does a ./mach configure. The code for check-java only runs an export. Does it need to do something else?

abpostelnicu commented 6 years ago

Why didn’t we integrate the configure in the check command, like we did with the other static-analysis targets?

rbartlensky commented 6 years ago

@abpostelnicu I followed the way check was implemented. The check command only does an export.

abpostelnicu commented 6 years ago

Yes because the export also calls the configure. You cannot run pre-export nor export without having the tree configured.

abpostelnicu commented 6 years ago

Another thing that troubles me here, looking at log i see this question mark:

Unzipping /cache/sa-unified/?/.gradle/wrapper/dists/gradle-4.4-all/9br9xq1tocpiv8o6njlyu5op1/gradle-4.4-all.zip to /cache/sa-unified/?/.gradle/wrapper/dists/gradle-4.4-all/9br9xq1tocpiv8o6njlyu5op1

How does that end up there? Probably because we don't set 'GRADLE_USER_HOME' like on so many tasks from TC, see https://searchfox.org/mozilla-central/source/taskcluster/ci/build/android.yml#657

abpostelnicu commented 6 years ago

@abpostelnicu I followed the way check was implemented. The check command only does an export.

Do you have an actual TC example where this analysis worked? I'm interested to see the difference on .gradle path. Until now the only idea that I have on why this is failing is because the boogie path that .gradle shows.

rbartlensky commented 6 years ago

Do you have an actual TC example where this analysis worked?

Unfortunately I don't. I will try to see what happens when I set GRADLE_HOME.

abpostelnicu commented 5 years ago

Before having that error, I've encountered other error regarding the artifact, infer is not located in the same namespace as the android artifacts so we must use a different mechanism here.

Running it locally there is a big difference on where gradle is setting it's local directory, locally we have it here: Unzipping /home/app/.gradle/wrapper/dists/gradle-4.4-all

But in the staging environment we get: Unzipping /cache/sa-unified/?/.gradle/wrapper/dists/gradle-4.4-all

Put despite this, locally I still cannot run it successfully:

BUILD SUCCESSFUL in 16s
6 actionable tasks: 4 executed, 2 up-to-date
/tmp/mozilla-state/infer/infer/bin/infer capture --skip-analysis-in-path browser/components/translation/cld2/ --skip-analysis-in-path browser/extensions/mortar/ppapi/ --skip-analysis-in-path db/sqlite3/src/ --skip-analysis-in-path devtools/client/sourceeditor/codemirror/ --skip-analysis-in-path devtools/client/sourceeditor/tern/ --skip-analysis-in-path devtools/shared/heapsnapshot/CoreDump.pb.cc --skip-analysis-in-path devtools/shared/heapsnapshot/CoreDump.pb.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_export.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --skip-analysis-in-path editor/libeditor/tests/browserscope/lib/richtext/ --skip-analysis-in-path editor/libeditor/tests/browserscope/lib/richtext2/ --skip-analysis-in-path extensions/spellcheck/hunspell/src/ --skip-analysis-in-path gfx/angle/ --skip-analysis-in-path gfx/cairo/ --skip-analysis-in-path gfx/graphite2/ --skip-analysis-in-path gfx/harfbuzz/ --skip-analysis-in-path gfx/ots/ --skip-analysis-in-path gfx/qcms/ --skip-analysis-in-path gfx/sfntly/ --skip-analysis-in-path gfx/skia/ --skip-analysis-in-path gfx/vr/openvr/ --skip-analysis-in-path gfx/webrender/ --skip-analysis-in-path gfx/webrender_api/ --skip-analysis-in-path gfx/wrench/ --skip-analysis-in-path gfx/ycbcr/ --skip-analysis-in-path intl/hyphenation/hyphen/ --skip-analysis-in-path intl/icu/ --skip-analysis-in-path ipc/chromium/ --skip-analysis-in-path js/src/ctypes/libffi/ --skip-analysis-in-path js/src/dtoa.c --skip-analysis-in-path js/src/jit/arm64/vixl/ --skip-analysis-in-path js/src/vtune/disable_warnings.h --skip-analysis-in-path js/src/vtune/ittnotify.h --skip-analysis-in-path js/src/vtune/ittnotify_config.h --skip-analysis-in-path js/src/vtune/ittnotify_static.c --skip-analysis-in-path js/src/vtune/ittnotify_static.h --skip-analysis-in-path js/src/vtune/ittnotify_types.h --skip-analysis-in-path js/src/vtune/jitprofiling.c --skip-analysis-in-path js/src/vtune/jitprofiling.h --skip-analysis-in-path js/src/vtune/legacy/ --skip-analysis-in-path media/ffvpx/ --skip-analysis-in-path media/gmp-clearkey/0.1/openaes/ --skip-analysis-in-path media/kiss_fft/ --skip-analysis-in-path media/libav/ --skip-analysis-in-path media/libcubeb/ --skip-analysis-in-path media/libjpeg/ --skip-analysis-in-path media/libmkv/ --skip-analysis-in-path media/libnestegg/ --skip-analysis-in-path media/libogg/ --skip-analysis-in-path media/libopus/ --skip-analysis-in-path media/libpng/ --skip-analysis-in-path media/libsoundtouch/ --skip-analysis-in-path media/libspeex_resampler/ --skip-analysis-in-path media/libtheora/ --skip-analysis-in-path media/libtremor/ --skip-analysis-in-path media/libvorbis/ --skip-analysis-in-path media/libvpx/ --skip-analysis-in-path media/libyuv/ --skip-analysis-in-path media/mtransport/third_party/ --skip-analysis-in-path media/openmax_dl/ --skip-analysis-in-path media/webrtc/signaling/src/sdp/sipcc/ --skip-analysis-in-path media/webrtc/trunk/ --skip-analysis-in-path mfbt/decimal/ --skip-analysis-in-path mfbt/double-conversion/double-conversion/ --skip-analysis-in-path mfbt/lz4.* --skip-analysis-in-path mobile/android/geckoview/src/thirdparty/ --skip-analysis-in-path mobile/android/thirdparty/ --skip-analysis-in-path modules/brotli/ --skip-analysis-in-path modules/fdlibm/ --skip-analysis-in-path modules/freetype2/ --skip-analysis-in-path modules/pdfium/ --skip-analysis-in-path modules/woff2/ --skip-analysis-in-path modules/xz-embedded/ --skip-analysis-in-path modules/zlib/ --skip-analysis-in-path netwerk/sctp/src/ --skip-analysis-in-path netwerk/srtp/src/ --skip-analysis-in-path nsprpub/ --skip-analysis-in-path other-licenses/ --skip-analysis-in-path parser/expat/ --skip-analysis-in-path security/nss/ --skip-analysis-in-path security/sandbox/chromium/ --skip-analysis-in-path testing/gtest/gmock/ --skip-analysis-in-path testing/gtest/gtest/ --skip-analysis-in-path testing/talos/talos/tests/dromaeo/ --skip-analysis-in-path testing/talos/talos/tests/kraken/ --skip-analysis-in-path testing/talos/talos/tests/v8_7/ --skip-analysis-in-path testing/web-platform/tests/resources/webidl2/ --skip-analysis-in-path third_party/aom/ --skip-analysis-in-path third_party/msgpack/ --skip-analysis-in-path third_party/prio/ --skip-analysis-in-path third_party/python/ --skip-analysis-in-path third_party/rust/ --skip-analysis-in-path toolkit/components/jsoncpp/ --skip-analysis-in-path toolkit/components/protobuf/ --skip-analysis-in-path toolkit/components/url-classifier/chromium/ --skip-analysis-in-path toolkit/components/url-classifier/protobuf/ --skip-analysis-in-path toolkit/crashreporter/google-breakpad/ --skip-analysis-in-path toolkit/recordreplay/udis86/ --skip-analysis-in-path tools/fuzzing/libfuzzer/ -- /app/tmp/sa-unified/gradlew compileLocalWithGeckoBinariesNoMinApiDebugSources
Error running mach:

    [\'--log-no-times\', \'static-analysis\', \'check-java\', \'mobile/android/geckoview/src/main/java/org/mozilla/gecko/GeckoAppShell.java\', \'.arcconfig\']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 2: [u\'/tmp/mozilla-state/infer/infer/bin/infer\', u\'capture\', u\'--skip-analysis-in-path\', u\'browser/components/translation/cld2/\', u\'--skip-analysis-in-path\', u\'browser/extensions/mortar/ppapi/\', u\'--skip-analysis-in-path\', u\'db/sqlite3/src/\', u\'--skip-analysis-in-path\', u\'devtools/client/sourceeditor/codemirror/\', u\'--skip-analysis-in-path\', u\'devtools/client/sourceeditor/tern/\', u\'--skip-analysis-in-path\', u\'devtools/shared/heapsnapshot/CoreDump.pb.cc\', u\'--skip-analysis-in-path\', u\'devtools/shared/heapsnapshot/CoreDump.pb.h\', u\'--skip-analysis-in-path\', u\'dom/media/gmp/widevine-adapter/content_decryption_module.h\', u\'--skip-analysis-in-path\', u\'dom/media/gmp/widevine-adapter/content_decryption_module_export.h\', u\'--skip-analysis-in-path\', u\'dom/media/gmp/widevine-adapter/content_decryption_module_ext.h\', u\'--skip-analysis-in-path\', u\'dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h\', u\'--skip-analysis-in-path\', u\'editor/libeditor/tests/browserscope/lib/richtext/\', u\'--skip-analysis-in-path\', u\'editor/libeditor/tests/browserscope/lib/richtext2/\', u\'--skip-analysis-in-path\', u\'extensions/spellcheck/hunspell/src/\', u\'--skip-analysis-in-path\', u\'gfx/angle/\', u\'--skip-analysis-in-path\', u\'gfx/cairo/\', u\'--skip-analysis-in-path\', u\'gfx/graphite2/\', u\'--skip-analysis-in-path\', u\'gfx/harfbuzz/\', u\'--skip-analysis-in-path\', u\'gfx/ots/\', u\'--skip-analysis-in-path\', u\'gfx/qcms/\', u\'--skip-analysis-in-path\', u\'gfx/sfntly/\', u\'--skip-analysis-in-path\', u\'gfx/skia/\', u\'--skip-analysis-in-path\', u\'gfx/vr/openvr/\', u\'--skip-analysis-in-path\', u\'gfx/webrender/\', u\'--skip-analysis-in-path\', u\'gfx/webrender_api/\', u\'--skip-analysis-in-path\', u\'gfx/wrench/\', u\'--skip-analysis-in-path\', u\'gfx/ycbcr/\', u\'--skip-analysis-in-path\', u\'intl/hyphenation/hyphen/\', u\'--skip-analysis-in-path\', u\'intl/icu/\', u\'--skip-analysis-in-path\', u\'ipc/chromium/\', u\'--skip-analysis-in-path\', u\'js/src/ctypes/libffi/\', u\'--skip-analysis-in-path\', u\'js/src/dtoa.c\', u\'--skip-analysis-in-path\', u\'js/src/jit/arm64/vixl/\', u\'--skip-analysis-in-path\', u\'js/src/vtune/disable_warnings.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/ittnotify.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/ittnotify_config.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/ittnotify_static.c\', u\'--skip-analysis-in-path\', u\'js/src/vtune/ittnotify_static.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/ittnotify_types.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/jitprofiling.c\', u\'--skip-analysis-in-path\', u\'js/src/vtune/jitprofiling.h\', u\'--skip-analysis-in-path\', u\'js/src/vtune/legacy/\', u\'--skip-analysis-in-path\', u\'media/ffvpx/\', u\'--skip-analysis-in-path\', u\'media/gmp-clearkey/0.1/openaes/\', u\'--skip-analysis-in-path\', u\'media/kiss_fft/\', u\'--skip-analysis-in-path\', u\'media/libav/\', u\'--skip-analysis-in-path\', u\'media/libcubeb/\', u\'--skip-analysis-in-path\', u\'media/libjpeg/\', u\'--skip-analysis-in-path\', u\'media/libmkv/\', u\'--skip-analysis-in-path\', u\'media/libnestegg/\', u\'--skip-analysis-in-path\', u\'media/libogg/\', u\'--skip-analysis-in-path\', u\'media/libopus/\', u\'--skip-analysis-in-path\', u\'media/libpng/\', u\'--skip-analysis-in-path\', u\'media/libsoundtouch/\', u\'--skip-analysis-in-path\', u\'media/libspeex_resampler/\', u\'--skip-analysis-in-path\', u\'media/libtheora/\', u\'--skip-analysis-in-path\', u\'media/libtremor/\', u\'--skip-analysis-in-path\', u\'media/libvorbis/\', u\'--skip-analysis-in-path\', u\'media/libvpx/\', u\'--skip-analysis-in-path\', u\'media/libyuv/\', u\'--skip-analysis-in-path\', u\'media/mtransport/third_party/\', u\'--skip-analysis-in-path\', u\'media/openmax_dl/\', u\'--skip-analysis-in-path\', u\'media/webrtc/signaling/src/sdp/sipcc/\', u\'--skip-analysis-in-path\', u\'media/webrtc/trunk/\', u\'--skip-analysis-in-path\', u\'mfbt/decimal/\', u\'--skip-analysis-in-path\', u\'mfbt/double-conversion/double-conversion/\', u\'--skip-analysis-in-path\', u\'mfbt/lz4.*\', u\'--skip-analysis-in-path\', u\'mobile/android/geckoview/src/thirdparty/\', u\'--skip-analysis-in-path\', u\'mobile/android/thirdparty/\', u\'--skip-analysis-in-path\', u\'modules/brotli/\', u\'--skip-analysis-in-path\', u\'modules/fdlibm/\', u\'--skip-analysis-in-path\', u\'modules/freetype2/\', u\'--skip-analysis-in-path\', u\'modules/pdfium/\', u\'--skip-analysis-in-path\', u\'modules/woff2/\', u\'--skip-analysis-in-path\', u\'modules/xz-embedded/\', u\'--skip-analysis-in-path\', u\'modules/zlib/\', u\'--skip-analysis-in-path\', u\'netwerk/sctp/src/\', u\'--skip-analysis-in-path\', u\'netwerk/srtp/src/\', u\'--skip-analysis-in-path\', u\'nsprpub/\', u\'--skip-analysis-in-path\', u\'other-licenses/\', u\'--skip-analysis-in-path\', u\'parser/expat/\', u\'--skip-analysis-in-path\', u\'security/nss/\', u\'--skip-analysis-in-path\', u\'security/sandbox/chromium/\', u\'--skip-analysis-in-path\', u\'testing/gtest/gmock/\', u\'--skip-analysis-in-path\', u\'testing/gtest/gtest/\', u\'--skip-analysis-in-path\', u\'testing/talos/talos/tests/dromaeo/\', u\'--skip-analysis-in-path\', u\'testing/talos/talos/tests/kraken/\', u\'--skip-analysis-in-path\', u\'testing/talos/talos/tests/v8_7/\', u\'--skip-analysis-in-path\', u\'testing/web-platform/tests/resources/webidl2/\', u\'--skip-analysis-in-path\', u\'third_party/aom/\', u\'--skip-analysis-in-path\', u\'third_party/msgpack/\', u\'--skip-analysis-in-path\', u\'third_party/prio/\', u\'--skip-analysis-in-path\', u\'third_party/python/\', u\'--skip-analysis-in-path\', u\'third_party/rust/\', u\'--skip-analysis-in-path\', u\'toolkit/components/jsoncpp/\', u\'--skip-analysis-in-path\', u\'toolkit/components/protobuf/\', u\'--skip-analysis-in-path\', u\'toolkit/components/url-classifier/chromium/\', u\'--skip-analysis-in-path\', u\'toolkit/components/url-classifier/protobuf/\', u\'--skip-analysis-in-path\', u\'toolkit/crashreporter/google-breakpad/\', u\'--skip-analysis-in-path\', u\'toolkit/recordreplay/udis86/\', u\'--skip-analysis-in-path\', u\'tools/fuzzing/libfuzzer/\', u\'--\', u\'/app/tmp/sa-unified/gradlew\', \'compileLocalWithGeckoBinariesNoMinApiDebugSources\']

  File "/app/tmp/sa-unified/python/mozbuild/mozbuild/mach_commands.py", line 1789, in check_java
    pass_thru=True)
  File "/app/tmp/sa-unified/python/mach/mach/mixin/process.py", line 162, in run_process
    raise Exception(\'Process executed with non-0 exit code %d: %s\' % (status, args))
'

abpostelnicu commented 5 years ago

This issue is because:

Uncaught Internal Error: ("InferBase__SqliteUtils.Error(\"exec: synchronous=OFF: IOERR (disk I/O error) (disk I/O error)\")")
Error backtrace:
Raised at file "base/SqliteUtils.ml" (inlined), line 13, characters 46-63
Called from file "format.ml", line 1364, characters 4-36
Called from file "base/SqliteUtils.ml" (inlined), line 27, characters 4-61
Called from file "base/ResultsDatabase.ml" (inlined), line 142, characters 2-75
Called from file "base/ResultsDir.ml" (inlined), line 52, characters 2-44
Called from file "base/ResultsDir.ml" (inlined), line 69, characters 2-27
Called from file "infer.ml" (inlined), line 45, characters 6-38
Called from file "infer.ml", line 116, characters 2-10

rbartlensky commented 5 years ago

This issue is because:

/tmp/mozilla-state/infer/infer/bin/infer: error while loading shared libraries: libsqlite3.so.0: cannot open shared object file: No such file or directory

It seems we miss libsqlite3, but this is strange since we have it in the nix environment.

If you run it manually, make sure you execute gecko-env your_command, because otherwise it will fail.

abpostelnicu commented 5 years ago

@rbartlensky do yo have any idea what the above error could be? Could it be a problem with the infer artifact that we are using, and that is not compatible with the nix env?

rbartlensky commented 5 years ago

@rbartlensky do yo have any idea what the above error could be?

Could you tell me the commands that you execute?

abpostelnicu commented 5 years ago

@rbartlensky please see:

/tmp/mozilla-state/infer/infer/bin/infer capture --skip-analysis-in-path browser/components/translation/cld2/ --skip-analysis-in-path browser/extensions/mortar/ppapi/ --skip-analysis-in-path db/sqlite3/src/ --skip-analysis-in-path devtools/client/sourceeditor/codemirror/ --skip-analysis-in-path devtools/client/sourceeditor/tern/ --skip-analysis-in-path devtools/shared/heapsnapshot/CoreDump.pb.cc --skip-analysis-in-path devtools/shared/heapsnapshot/CoreDump.pb.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_export.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_ext.h --skip-analysis-in-path dom/media/gmp/widevine-adapter/content_decryption_module_proxy.h --skip-analysis-in-path editor/libeditor/tests/browserscope/lib/richtext/ --skip-analysis-in-path editor/libeditor/tests/browserscope/lib/richtext2/ --skip-analysis-in-path extensions/spellcheck/hunspell/src/ --skip-analysis-in-path gfx/angle/ --skip-analysis-in-path gfx/cairo/ --skip-analysis-in-path gfx/graphite2/ --skip-analysis-in-path gfx/harfbuzz/ --skip-analysis-in-path gfx/ots/ --skip-analysis-in-path gfx/qcms/ --skip-analysis-in-path gfx/sfntly/ --skip-analysis-in-path gfx/skia/ --skip-analysis-in-path gfx/vr/openvr/ --skip-analysis-in-path gfx/webrender/ --skip-analysis-in-path gfx/webrender_api/ --skip-analysis-in-path gfx/wrench/ --skip-analysis-in-path gfx/ycbcr/ --skip-analysis-in-path intl/hyphenation/hyphen/ --skip-analysis-in-path intl/icu/ --skip-analysis-in-path ipc/chromium/ --skip-analysis-in-path js/src/ctypes/libffi/ --skip-analysis-in-path js/src/dtoa.c --skip-analysis-in-path js/src/jit/arm64/vixl/ --skip-analysis-in-path js/src/vtune/disable_warnings.h --skip-analysis-in-path js/src/vtune/ittnotify.h --skip-analysis-in-path js/src/vtune/ittnotify_config.h --skip-analysis-in-path js/src/vtune/ittnotify_static.c --skip-analysis-in-path js/src/vtune/ittnotify_static.h --skip-analysis-in-path js/src/vtune/ittnotify_types.h --skip-analysis-in-path js/src/vtune/jitprofiling.c --skip-analysis-in-path js/src/vtune/jitprofiling.h --skip-analysis-in-path js/src/vtune/legacy/ --skip-analysis-in-path media/ffvpx/ --skip-analysis-in-path media/gmp-clearkey/0.1/openaes/ --skip-analysis-in-path media/kiss_fft/ --skip-analysis-in-path media/libav/ --skip-analysis-in-path media/libcubeb/ --skip-analysis-in-path media/libjpeg/ --skip-analysis-in-path media/libmkv/ --skip-analysis-in-path media/libnestegg/ --skip-analysis-in-path media/libogg/ --skip-analysis-in-path media/libopus/ --skip-analysis-in-path media/libpng/ --skip-analysis-in-path media/libsoundtouch/ --skip-analysis-in-path media/libspeex_resampler/ --skip-analysis-in-path media/libtheora/ --skip-analysis-in-path media/libtremor/ --skip-analysis-in-path media/libvorbis/ --skip-analysis-in-path media/libvpx/ --skip-analysis-in-path media/libyuv/ --skip-analysis-in-path media/mtransport/third_party/ --skip-analysis-in-path media/openmax_dl/ --skip-analysis-in-path media/webrtc/signaling/src/sdp/sipcc/ --skip-analysis-in-path media/webrtc/trunk/ --skip-analysis-in-path mfbt/decimal/ --skip-analysis-in-path mfbt/double-conversion/double-conversion/ --skip-analysis-in-path mfbt/lz4.* --skip-analysis-in-path mobile/android/geckoview/src/thirdparty/ --skip-analysis-in-path mobile/android/thirdparty/ --skip-analysis-in-path modules/brotli/ --skip-analysis-in-path modules/fdlibm/ --skip-analysis-in-path modules/freetype2/ --skip-analysis-in-path modules/pdfium/ --skip-analysis-in-path modules/woff2/ --skip-analysis-in-path modules/xz-embedded/ --skip-analysis-in-path modules/zlib/ --skip-analysis-in-path netwerk/sctp/src/ --skip-analysis-in-path netwerk/srtp/src/ --skip-analysis-in-path nsprpub/ --skip-analysis-in-path other-licenses/ --skip-analysis-in-path parser/expat/ --skip-analysis-in-path security/nss/ --skip-analysis-in-path security/sandbox/chromium/ --skip-analysis-in-path testing/gtest/gmock/ --skip-analysis-in-path testing/gtest/gtest/ --skip-analysis-in-path testing/talos/talos/tests/dromaeo/ --skip-analysis-in-path testing/talos/talos/tests/kraken/ --skip-analysis-in-path testing/talos/talos/tests/v8_7/ --skip-analysis-in-path testing/web-platform/tests/resources/webidl2/ --skip-analysis-in-path third_party/aom/ --skip-analysis-in-path third_party/msgpack/ --skip-analysis-in-path third_party/prio/ --skip-analysis-in-path third_party/python/ --skip-analysis-in-path third_party/rust/ --skip-analysis-in-path toolkit/components/jsoncpp/ --skip-analysis-in-path toolkit/components/protobuf/ --skip-analysis-in-path toolkit/components/url-classifier/chromium/ --skip-analysis-in-path toolkit/components/url-classifier/protobuf/ --skip-analysis-in-path toolkit/crashreporter/google-breakpad/ --skip-analysis-in-path toolkit/recordreplay/udis86/ --skip-analysis-in-path tools/fuzzing/libfuzzer/ -- /app/tmp/sa-unified/gradlew compileLocalWithGeckoBinariesNoMinApiDebugSources

Of course in gecko-env, I did this because this was the step of mach that is failing.

rbartlensky commented 5 years ago

@abpostelnicu Try to write the above command in a file (let's call it infer.sh), and then run gecko-env infer.sh

abpostelnicu commented 5 years ago

This seems to be blocked since I see here two issues:

[x] Original issue that replicates on TC, and most likely is caused by the lack of GRADLE_HOME, that's why we see in the paths the strange ?.
[x] The most recent issue that impairs us on running infer capture on a local nix instance.

I will address here only the 2nd issue since in the current state this one blocks the original issue for which this bug has been opened. The problem that we encounter is from the first step of the analysis when we try to run:

infer capture ...

This fails with:

Uncaught Internal Error: ("InferBase__SqliteUtils.Error(\"exec: synchronous=OFF: IOERR (disk I/O error) (disk I/O error)\")") Error backtrace: Raised at file "base/SqliteUtils.ml" (inlined), line 13, characters 46-63 Called from file "format.ml", line 1364, characters 4-36 Called from file "base/SqliteUtils.ml" (inlined), line 27, characters 4-61 Called from file "base/ResultsDatabase.ml" (inlined), line 142, characters 2-75 Called from file "base/ResultsDir.ml" (inlined), line 52, characters 2-44 Called from file "base/ResultsDir.ml" (inlined), line 69, characters 2-27 Called from file "infer.ml" (inlined), line 45, characters 6-38 Called from file "infer.ml", line 116, characters 2-10

This can be reproduced in the nix environment outside of the analysis by simply running: /tmp/mozilla-state/infer/infer/bin/infer capture -- /app/tmp/sa-unified/gradlew compileLocalWithGeckoBinariesNoMinApiDebugSources

Looking at the stack-trace from the exception it seems there is a problem with the sqlite library that we use, but I'm unaware on how to pursue this further. Until now I've used two infer binaries, the official one and the one that we provided from the artifact toolchain, both of them have the same outcome.

abpostelnicu commented 5 years ago

The issue related with infer capture and sqlite is fixed, it was something that only locally was affecting the build due to communication timeout between sqlite driver and the underlying docker image. This was due because the infer/ output directory was shared with the host, causing writing delay.

Now coming back to the first issue, it can only be reproduced on tc and i will try a push with GRADLE_USER_HOME set.

marco-c commented 5 years ago

The issue related with infer capture and sqlite is fixed, it was something that only locally was affecting the build due to communication timeout between sqlite driver and the underlying docker image. This was due because the infer/ output directory was shared with the host, causing writing delay.

I feel your pain :P

abpostelnicu commented 5 years ago

In order to fix [1] I've created this PR.

abpostelnicu commented 5 years ago

The problem here is more complex, the gradle home issue was generated by Java, that when calling System.getProperty("user.home") returns ?. This is a known OpenJDK bug on no-so-standard environments.

All of this is mitigated by this PR.

Triggering a test run with: export _JAVA_OPTIONS="-Duser.home=\"/tmp/mozilla-state\"" the analysis fails because infer crashes with:

Uncaught Internal Error: (Sys_error "/etc/localtime: No such file or directory") Error backtrace: Raised by primitive operation at file "pervasives.ml", line 389, characters 28-54 Called from file "src/in_channel.ml" (inlined), line 17, characters 2-35 Called from file "src/zone.ml", line 343, characters 21-49 Called from file "src/core_zone.ml", line 160, characters 6-73 Called from file "camlinternalLazy.ml", line 27, characters 17-27 Re-raised at file "camlinternalLazy.ml", line 34, characters 4-11 Called from file "src/core_time.ml" (inlined), line 210, characters 42-65 Called from file "base/RunState.ml", line 10, characters 22-51

This is because on TC we don't expose to the container /etc/localtime. Hopefully once we do this we will have everything up and running.

abpostelnicu commented 5 years ago

As this has been fixed in 1632. Now we are encountering: Uncaught Internal Error: (Sys_error "/usr/share/zoneinfo/: No such file or directory")

abpostelnicu commented 5 years ago

With the latest fixes we are at the stage where our task no longer fails. This is great but infer doesn't find any sources that it can work with in infer-out.

This last issue can be reproduced locally very easy inside mach.

rbartlensky commented 5 years ago

With the latest fixes we are at the stage where our task no longer fails. This is great but infer doesn't find any sources that it can work with in infer-out.

Is this a fresh task? If so, I find it quite odd that it does not find any issues. Does it find issues locally?

abpostelnicu commented 5 years ago

With the latest fixes we are at the stage where our task no longer fails. This is great but infer doesn't find any sources that it can work with in infer-out.

Is this a fresh task? If so, I find it quite odd that it does not find any issues. Does it find issues locally?

This is a fresh task. This can be reproved locally if we ran infer via mach.

rbartlensky commented 5 years ago

Why is python/mozbuild/mozbuild/mach_commands.py included in check-java? If you run mach I think it will output the command that it executes at each step. What happens if you execute those manually?

abpostelnicu commented 5 years ago

What do you mean by your last comment regarding mach_commands.py? I think the problem lies that we create a tmp file and we put in that file the files that we want to be scanned. For example I'm doing the static-analysis for: mobile/android/app/src/photon/java/org/mozilla/gecko/home/SuggestionItem.java

So i can reproduce the problem if i do: infer analyze --changed-files-index ~/Desktop/file Whereas in that file from Desktop I have mobile/android/app/src/photon/java/org/mozilla/gecko/home/SuggestionItem.java

rbartlensky commented 5 years ago

static_analysis_bot.infer.infer: Running static-analysis (cmd='gecko-env ./mach --log-no-times static-analysis check-java python/mozbuild/mozbuild/mach_commands.py mobile/android/app/src/photon/java/org/mozilla/gecko/home/SuggestionItem.java .arcconfig')

This means that mach_commands has been modified, which I find strange.

I think the problem lies that we create a tmp file and we put in that file the files that we want to be scanned.

The files are placed in a tmp file because the capturing is done for all files. The infer analyze command has an argument changed-files-index with which you can specify a file (the tmp file in our case) which contains all the files that should be analyzed.

abpostelnicu commented 5 years ago

OK so I've nailed this, the file that I wanted to test was not captured by infer during infer capture. But i found this to be very odd.

abpostelnicu commented 5 years ago

Good news it worked on staging.

abpostelnicu commented 5 years ago

This PR is the last fix for this to work. It has also been tested here.

abpostelnicu commented 5 years ago

This can be closed since every dependency has been resolved!

mozilla / release-services

Finish Infer Integration #1507

1: Task failed with an exception.

Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

2: Task failed with an exception.

Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.