search

aquasecurity / btfhub

BTFhub, in collaboration with the BTFhub Archive repository, supplies BTF files for all published kernels that lack native support for embedded BTF. This joint effort ensures that even kernels without built-in BTF support can effectively leverage the benefits of eBPF programs, promoting compatibility across various kernel versions.
Apache License 2.0
364 stars 42 forks source link

Duplicated BTFs files #17

Open mauriciovasquezbernal opened 2 years ago

mauriciovasquezbernal commented 2 years ago

This is rather not an issue but something that could be improved.

I found out that there are some BTF files with the exact same content for different kernel versions:

$ fdupes -r ./archive/ 
./archive/centos/8/x86_64/4.18.0-147.3.1.el8_1.x86_64.btf
./archive/centos/8/x86_64/4.18.0-147.5.1.el8_1.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1062.4.2.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1062.4.3.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1160.2.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1160.2.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1160.24.1.el7.x86_64.failed
./archive/centos/8/x86_64/4.18.0-240.el8.x86_64.failed

./archive/centos/7/x86_64/3.10.0-1160.41.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1160.45.1.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-327.36.2.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-327.36.3.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-327.3.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-327.4.4.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1062.1.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1062.1.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-693.1.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-693.2.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-514.6.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-514.6.2.el7.x86_64.btf

./archive/ubuntu/18.04/x86_64/4.15.0-106-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-108-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1020-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1022-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1047-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1048-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1050-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1033-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1034-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-115-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-117-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1014-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1018-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1045-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1047-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1022-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1024-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-121-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-122-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-123-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-124-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-109-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-111-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-112-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1103-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1104-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-69-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-70-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-129-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-130-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-132-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-134-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1034-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1035-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-24-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-29-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1028-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1030-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-40-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-42-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1028-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1029-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-45-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-47-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1039-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1040-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-51-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-52-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-53-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-54-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1093-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1095-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1077-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1079-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1086-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1087-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-156-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-158-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1023-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1025-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-74-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-76-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1080-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1082-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1059-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1061-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1073-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1076-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1018-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1020-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1096-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1097-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1091-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1092-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-84-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-86-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-44-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-45-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-139-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-140-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1057-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1058-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1110-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1111-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1035-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1036-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-60-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-62-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-64-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1048-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1049-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1011-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1016-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-151-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-153-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1099-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1101-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-59-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-60-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-62-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-64-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-65-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1020-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1022-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-26-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-28-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-29-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1045-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1047-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1022-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1024-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1015-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1017-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1034-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1035-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-40-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-42-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-45-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-47-generic.btf

./archive/ubuntu/20.04/x86_64/5.8.0-1040-azure.btf
./archive/ubuntu/20.04/x86_64/5.8.0-1041-azure.btf
./archive/ubuntu/20.04/x86_64/5.8.0-1042-azure.btf

./archive/ubuntu/20.04/x86_64/5.8.0-25-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-28-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-29-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-52-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-53-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-54-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1023-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1025-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-31-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-33-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1059-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1061-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1018-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1020-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1021-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-84-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-86-generic.btf

./archive/ubuntu/20.04/x86_64/5.8.0-34-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-36-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-38-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-40-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-41-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-43-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1016-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1019-azure.btf

./archive/ubuntu/20.04/x86_64/5.11.0-1017-azure.btf
./archive/ubuntu/20.04/x86_64/5.11.0-1019-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1035-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1036-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1048-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1049-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-37-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-39-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-59-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-60-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-62-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-64-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-65-generic.btf

I think this could be optimized to reduce the storage needs by replacing the duplicated ones with a symlink. The feasibility and importance of this optimization is related to the adopted storage solution -> https://github.com/aquasecurity/btfhub/issues/12.

rafaeldtinoco commented 2 years ago

I believe this can't currently be done because a symlink to the files would make the raw permlink not to work if someone tries to get it from github URL directly (which is a current use case), so I'm closing this, please do reopen if you see fit/need.

itaysk commented 2 years ago

@rafaeldtinoco are you sure about that? here's a test where the file bar is a symlink to the file foo and both work through GitHub raw urls: https://github.com/itaysk/actionstest https://raw.githubusercontent.com/itaysk/actionstest/main/foo https://raw.githubusercontent.com/itaysk/actionstest/main/bar