Closed antoniozh closed 1 year ago
It seems like that the data-acquisition-tool struggles to get a Go file inside folders of the project.
antoniozhu@WS0075-21:/mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit/dummy_project/dummy_package$ ls -d $PWD/*
/mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit/dummy_project/dummy_package/main.go
antoniozhu@WS0075-21:/mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit/dummy_project/dummy_package$ docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build -v /mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit:/projects usgoc/pred:latest --project dummy_project --line 11 --package dummy_package --file main.go predict -m WL2GNN
panic: Could not find non-cache file.
goroutine 1 [running]:
github.com/stg-tud/unsafe_go_study_results/scripts/data-acquisition-tool/cfg.GetCFG(0xc000121d48, {0x7ffeb335fe72, 0xc000121d78})
/app/cfg/cfg.go:701 +0x878
github.com/stg-tud/unsafe_go_study_results/scripts/data-acquisition-tool/cmd.glob..func3(0xed4c20, {0xa4adf7, 0xc, 0xc})
/app/cmd/cfg.go:23 +0xd9
github.com/spf13/cobra.(*Command).execute(0xed4c20, {0xc00014a0c0, 0xc, 0xc})
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xed56a0)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
github.com/stg-tud/unsafe_go_study_results/scripts/data-acquisition-tool/cmd.Execute()
/app/cmd/root.go:21 +0x25
main.main()
/app/main.go:8 +0x17
Traceback (most recent call last):
File "src/usgoc/run_prediction.py", line 147, in <module>
cli(obj=dict())
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "src/usgoc/run_prediction.py", line 117, in predict
cfg = get_cfg_json(**obj)
File "src/usgoc/run_prediction.py", line 36, in get_cfg_json
encoding="utf-8")
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/data-acquisition-tool', 'cfg', '--base', '/projects', '--project', 'dummy_project', '--package', 'dummy_package', '--file', 'main.go', '--line', '11', '--snippet', '']' returned non-zero exit status 2.
When I move the file to the root directory of the project, it works:
antoniozhu@WS0075-21:/mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit/dummy_project/dummy_package$ mv main.go ..
antoniozhu@WS0075-21:/mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit/dummy_project/dummy_package$ docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build -v /mnt/c/Users/antonio.zhu/Documents/repos/praktikum/unsafe-toolkit:/projects usgoc/pred:latest --project dummy_project --line 11 --package dummy_package --file main.go predict -m WL2GNN
2022-02-10 17:52:47.688147: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/core.py:1395: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.
Instructions for updating:
The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.
2022-02-10 17:52:48.393910: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2022-02-10 17:52:48.394238: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2495995000 Hz
[{"cast-basic": 8.219590199587401e-07, "cast-bytes": 1.8294153747433484e-08, "cast-header": 9.329493195764371e-07, "cast-pointer": 6.143737323327514e-07, "cast-struct": 1.0154550182051025e-05, "definition": 0.00011404613178456202, "delegate": 0.9998230338096619, "memory-access": 2.007181137742009e-05, "pointer-arithmetic": 2.608248541946523e-05, "syscall": 3.7869078823860036e-06, "unused": 4.3619456846499816e-07}, {"atomic": 0.0034588424023240805, "efficiency": 7.757153070997447e-06, "ffi": 0.9953190684318542, "generics": 0.00041620503179728985, "hide-escape": 2.2497886675409973e-05, "layout": 0.0005000202800147235, "no-gc": 4.292558878660202e-05, "reflect": 4.836152129428228e-06, "serialization": 0.0002216835127910599, "types": 2.287283450641553e-06, "unused": 3.897362603311194e-06}]
Hi! This issue is caused by the fact that the CFG generator retrieves the source files of a package via Go's package.Load
function. This means that the string provided via the --package
CLI argument must be a valid fully-qualified Go package name (i.e. using the same format that is also used in Go's import
statement).
For your particular issue, the problem should therefore be fixable by using --project protobuf@v1.25.0 --package google.golang.org/protobuf/internal/strs
with the volume mount -v /mnt/c/Users/antonio.zhu/go/pkg/mod/google.golang.org:/projects
.
In case the way project lookup works is unclear: It does not matter how you partition the project path between the /projects
volume-mount and the --project
argument, i.e. -v /a/b:/projects
+ --project c
is completely equivalent to -v /a:/projects
+ --project b/c
. Also, note that the combined project dir (/projects
mountpoint concatenated with the --project
argument) should typically always contain a go.mod
file; elsewise the fully-qualified module name for the packages within the project and the version numbers for project-external package dependencies are unknown.
I hope this helps. Let me know if you need any further information. :slightly_smiling_face:
Hi, this works! Since I generate the arguments with Python, would you know of any way the go-geiger output could be used to parse the fully qualified package name? Is the fully qualified package name the path of the package relative to the projects folder path?
Great! Since I'm not very familiar with the go-geiger tool, I'm not sure what the easiest way to obtain the full package name is; maybe @gh0st42 or @akwick have an idea. 🙂
Alternatively, as you suggested, deriving the package name from the path in go/pkg/mod
should probably also be possible. With this approach you would have to remove the version tags from module directories (e.g. the @v1.25.0
part).
Hi, there are some exceptions to the fully qualified package name. For example when I build the following project and try to analyze the files the package name is one that I would not find in the file system. Is there any other way to find out the package name for a Go file?
This is the generated command:
docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build usgoc/pred:latest --project crypto@v0.0.0-20220215181150-74469fa99b22 --line 18 --package github.com/drakkan/crypto/internal/subtle --file aliasing.go --base /root/go/pkg/mod/github.com/drakkan predict -m WL2GNN
This is the fixed command, I found the correct package name in a comment in the aliasing.go file.
docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build usgoc/pred:latest --project crypto@v0.0.0-20220215181150-74469fa99b22 --line 18 --package golang.org/x/crypto/internal/subtle --file aliasing.go --base /root/go/pkg/mod/github.com/drakkan predict -m WL2GNN
Yes, good point. I hadn't thought about forks. The path-based approach to derive the package name of a file is not sufficient then.
If you can't get the package name from the go-geiger tool, getting the module name from the go.mod
file in the --project
dir is probably the easiest solution (assuming, of course, that the file with the unsafe usage always belongs to the project and is not part of some dependency of it). A simple Regex matching ^module (.+)
might already be sufficient.
I tried to classify a line in a go file found by go-geiger and a critical error occured. Here is how I recreate the error:
My script gave me the following docker command:
docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build -v /mnt/c/Users/antonio.zhu/go/pkg/mod/google.golang.org/protobuf@v1.25.0/internal:/projects usgoc/pred:latest --project strs --line 17 --package strs --file strings_unsafe.go predict -m WL2GNN
But after checking the arguments, I changed them to the following arguments.
docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build -v /mnt/c/Users/antonio.zhu/go/pkg/mod/google.golang.org:/projects usgoc/pred:latest --project protobuf@v1.25.0 --line 17 --package strs --file strings_unsafe.go predict -m WL2GNN
I thought that the project should be pointing to the projects folder containing the go.mod file. Both commands did not work and I got the non-cache file error, so I figured that I should maybe add the relative file path to the file argument too.
docker run --rm -v go_mod:/root/go/pkg/mod -v go_cache:/root/.cache/go-build -v /mnt/c/Users/antonio.zhu/go/pkg/mod/google.golang.org:/projects usgoc/pred:latest --project protobuf@v1.25.0 --line 17 --package strs --file internal/strs/strings_unsafe.go predict -m WL2GNN
But all commands resulted in the following error: