search

fluent / fluent-package-builder

td-agent (Fluentd) Building and Packaging System
Apache License 2.0
22 stars 25 forks source link

fluent-package v5.1.0 #660

Closed kenhys closed 1 month ago

kenhys commented 1 month ago

Tracking v5.1.0 issues still in development.

kenhys commented 1 month ago

bundled gem size was increased. (~5MB)

5 0 4

5 1 0

kenhys commented 1 month ago

mainly rdkafka upgrade affect it.( 2.3MB=>7.2MB )

kenhys commented 1 month ago

need to relax limitation for a while.

kenhys commented 1 month ago

ffi 1.16.3 can't be built on AmazonLinux 2.

https://github.com/fluent/fluent-package-builder/actions/runs/10034848340/job/27729984596

kenhys commented 1 month ago

rdkafka 0.16.1 is not correctly built?

kenhys commented 1 month ago

If rdkafka was reverted to 0.16.0, package size was not so increased. (may be no need to relax limitation )

kenhys commented 1 month ago

rdkafka 0.16.0 is buldable, but not available in colfluent test.

kenhys commented 1 month ago

/opt/fluent/bin/gem list |grep rdkafka rdkafka (0.16.0)

rdkafka was installed. but not recognized... :thinking:

kenhys commented 1 month ago

rdkafka 0.16.0 was installed at:

~$ find /opt/fluent/ -name '*rdkafka*'
/opt/fluent/lib/ruby/gems/3.2.0/specifications/rdkafka-0.16.0.gemspec
/opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.16.0
/opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.16.0/lib/rdkafka
/opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.16.0/lib/rdkafka.rb
/opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.16.0/ext/librdkafka.so
/opt/fluent/lib/ruby/gems/3.2.0/gems/rdkafka-0.16.0/rdkafka.gemspec
/opt/fluent/lib/ruby/gems/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/in_rdkafka_group.rb
/opt/fluent/lib/ruby/gems/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka.rb
/opt/fluent/lib/ruby/gems/3.2.0/gems/fluent-plugin-kafka-0.19.2/lib/fluent/plugin/out_rdkafka2.rb
/opt/fluent/lib/ruby/gems/3.2.0/extensions/x86_64-linux/3.2.0/rdkafka-0.16.0
kenhys commented 1 month ago

The rdkafka failure is caused by removing gem.build_complete.

kenhys commented 1 month ago

https://github.com/fluent/fluent-package-builder/actions/runs/10036365251/job/27733978175

windows_platforms = [:mingw, :x64_mingw] # :mswin
gem "ffi", "1.16.3", platforms: windows_platforms

Thus, platforms: does not work as expected on AmazonLinux 2.

kenhys commented 1 month ago 
+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
Compiling /root/rpmbuild/BUILDROOT/fluent-package-5.1.0-1.amzn2.x86_64/opt/fluent/lib/ruby/gems/3.2.0/gems/ffi-1.16.3/ext/ffi_c/libffi/generate-darwin-source-and-headers.py ...

And during building debuinfo?, unexpectedly bytecompile process was kicked.

kenhys commented 1 month ago

And during building debuinfo?, unexpectedly bytecompile process was kicked.

Same result (v1.17.0) on AmazonLinux 2.

kenhys commented 1 month ago

https://fedoraproject.org/wiki/Changes/No_more_automagic_Python_bytecompilation

does not work as expected...

kenhys commented 1 month ago

We should follow https://docs.fedoraproject.org/en-US/packaging-guidelines/#_brp_buildroot_policy_scripts

kenhys commented 1 month ago

container ea60ff1bf4515e0721f4583107bf62c2fee67b8fe97b9435ff8ca5bf4d1a295c encountered an error during hcs::System::Start: context deadline exceeded

it may be resolved with retry.

kenhys commented 1 month ago

Check whether rdkafka 0.16.0 => 0.16.1 upgrade can be bundled or not again. (as gem build_complete issue was resolved)

kenhys commented 1 month ago

Blocker issue on Windows:

Downloading fluent-plugin-webhdfs-1.6.0 revealed dependencies not in the API or 
the lockfile (webhdfs (>= 0.11.0)).
Either installing with `--full-index` or running `bundle update
fluent-plugin-webhdfs` should fix the problem.
rake aborted!
kenhys commented 1 month ago

Note: confluent community can't be installed on AmazonLinux (via package management)

kenhys commented 1 month ago

Note: confluent community can't be installed on AmazonLinux (via package management)

Precisely, confluent-client repository is not supported.

kenhys commented 1 month ago

build failure on windows was fixed in https://github.com/fluent/fluent-package-builder/actions/runs/10052696903/job/27784298154

kenhys commented 1 month ago

kafka related test case is bit out-dated. so refactor it now.

kenhys commented 1 month ago

NOTE:

Now macOS workflow always fails because of refreshed runner image

https://github.com/fluent/fluent-package-builder/actions/runs/10105868174/job/27947051208 macos-14-arm64 20240722.3 causes failure.

https://github.com/fluent/fluent-package-builder/actions/runs/10091587427/job/27903442530 until version macos-14-arm64 20240714.2, it succeeds.

kenhys commented 1 month ago

recently nokogiri 1.16.7 was released, catch up it.

kenhys commented 1 month ago

recently nokogiri 1.16.7 was released, catch up it.

done.

kenhys commented 1 month ago

We need to tackle windows build failure...

kenhys commented 1 month ago

Try to update to ruby 3.2.5.

kenhys commented 1 month ago

We need to tackle windows build failure...

Noticed that nokogiri 1.16.7 fails to build on windows, but not reproducible now...

kenhys commented 1 month ago

build process on arm works as expected.

kenhys commented 1 month ago

Even though some fixes for script are needed, but it works well with switching signing key. checked with test/experimental/5 repository.

kenhys commented 1 month ago

Done!