Closed ur5us closed 2 years ago
Getting the following compiler output:
rustc --version rustc 1.61.0 (fe5b13d68 2022-05-18) gem install tokenizers Building native extensions. This could take a while... ERROR: Error installing tokenizers: ERROR: Failed to build gem native extension. current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers /home/ur5us/.rvm/rubies/ruby-3.0.4/bin/ruby -I /home/ur5us/.rvm/rubies/ruby-3.0.4/lib/ruby/site_ruby/3.0.0 -r ./siteconf20220623-143041-cdpust.rb extconf.rb current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers make DESTDIR\= clean cargo clean current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers make DESTDIR\= cargo build --release Compiling libc v0.2.121 Compiling cfg-if v1.0.0 Compiling autocfg v1.1.0 Compiling cc v1.0.73 Compiling pkg-config v0.3.24 Compiling proc-macro2 v1.0.36 Compiling unicode-xid v0.2.2 Compiling syn v1.0.89 Compiling memchr v2.3.4 Compiling lazy_static v1.4.0 Compiling log v0.4.14 Compiling version_check v0.9.4 Compiling pin-project-lite v0.2.8 Compiling bitflags v1.3.2 Compiling futures-core v0.3.21 Compiling bytes v1.1.0 Compiling once_cell v1.10.0 Compiling itoa v1.0.1 Compiling serde_derive v1.0.136 Compiling futures-task v0.3.21 Compiling crossbeam-utils v0.8.8 Compiling typenum v1.15.0 Compiling futures-util v0.3.21 Compiling foreign-types-shared v0.1.1 Compiling serde v1.0.136 Compiling fnv v1.0.7 Compiling openssl v0.10.38 Compiling ryu v1.0.9 Compiling pin-utils v0.1.0 Compiling hashbrown v0.11.2 Compiling slab v0.4.5 Compiling crc32fast v1.3.2 Compiling unicode-width v0.1.9 Compiling futures-channel v0.3.21 Compiling native-tls v0.2.8 Compiling futures-io v0.3.21 Compiling tinyvec_macros v0.1.0 Compiling matches v0.1.9 Compiling httparse v1.6.0 Compiling futures-sink v0.3.21 Compiling scopeguard v1.1.0 Compiling radium v0.5.3 Compiling percent-encoding v2.1.0 Compiling strsim v0.9.3 Compiling ident_case v1.0.1 Compiling adler v1.0.2 Compiling rayon-core v1.9.1 Compiling getrandom v0.1.16 Compiling ppv-lite86 v0.2.16 Compiling try-lock v0.2.3 Compiling regex-syntax v0.6.25 Compiling openssl-probe v0.1.5 Compiling httpdate v1.0.2 Compiling lexical-core v0.7.6 Compiling unicode-bidi v0.3.7 Compiling encoding_rs v0.8.30 Compiling tower-service v0.3.1 Compiling either v1.6.1 Compiling byteorder v1.4.3 Compiling arrayvec v0.5.2 Compiling tap v1.0.1 Compiling wyz v0.2.0 Compiling static_assertions v1.1.0 Compiling serde_json v1.0.79 Compiling funty v1.1.0 Compiling fastrand v1.7.0 Compiling cpufeatures v0.2.2 Compiling mime v0.3.16 Compiling base64 v0.13.0 Compiling remove_dir_all v0.5.3 Compiling derive_builder v0.9.0 Compiling ipnet v2.4.0 Compiling number_prefix v0.4.0 Compiling ansi_term v0.12.1 Compiling glob v0.3.0 Compiling unicode-segmentation v1.9.0 Compiling strsim v0.8.0 Compiling macro_rules_attribute-proc_macro v0.0.2 Compiling smallvec v1.8.0 Compiling number_prefix v0.3.0 Compiling rutie v0.8.3 Compiling vec_map v0.8.2 Compiling base64 v0.12.3 Compiling unicode_categories v0.1.1 Compiling paste v1.0.6 Compiling tracing-core v0.1.23 Compiling indexmap v1.8.0 Compiling memoffset v0.6.5 Compiling crossbeam-epoch v0.9.8 Compiling miniz_oxide v0.4.4 Compiling rayon v1.5.1 Compiling generic-array v0.14.5 Compiling nom v6.2.1 Compiling foreign-types v0.3.2 Compiling http v0.2.6 Compiling textwrap v0.11.0 Compiling openssl-sys v0.9.72 Compiling bzip2-sys v0.1.11+1.0.8 Compiling onig_sys v69.7.1 Compiling esaxx-rs v0.1.7 Compiling tinyvec v1.5.1 Compiling form_urlencoded v1.0.1 Compiling itertools v0.8.2 Compiling itertools v0.9.0 Compiling unicode-normalization-alignments v0.1.12 Compiling macro_rules_attribute v0.0.2 Compiling tracing v0.1.32 Compiling aho-corasick v0.7.15 Compiling unicode-normalization v0.1.19 Compiling http-body v0.4.4 Compiling num_cpus v1.13.1 Compiling socket2 v0.4.4 Compiling terminal_size v0.1.17 Compiling getrandom v0.2.5 Compiling time v0.1.43 Compiling filetime v0.2.15 Compiling xattr v0.2.2 Compiling atty v0.2.14 Compiling tempfile v3.3.0 Compiling dirs-sys v0.3.7 Compiling fs2 v0.4.3 Compiling quote v1.0.16 Compiling mio v0.8.2 Compiling want v0.3.0 Compiling crossbeam-channel v0.5.4 Compiling bitvec v0.19.6 Compiling regex v1.4.6 Compiling idna v0.2.3 Compiling rand_core v0.6.3 Compiling rand_core v0.5.1 Compiling tar v0.4.38 Compiling clap v2.34.0 Compiling dirs v3.0.2 Compiling tokio v1.17.0 Compiling flate2 v1.0.22 Compiling block-buffer v0.10.2 Compiling crypto-common v0.1.3 Compiling rand_chacha v0.3.1 Compiling rand_chacha v0.2.2 Compiling url v2.2.2 Compiling bzip2 v0.4.3 Compiling console v0.15.0 Compiling crossbeam-deque v0.8.1 Compiling digest v0.10.3 Compiling rand v0.8.5 Compiling rand v0.7.3 Compiling tokio-util v0.6.9 Compiling indicatif v0.16.2 Compiling indicatif v0.15.0 Compiling darling_core v0.10.2 Compiling onig v6.3.1 Compiling sha2 v0.10.2 Compiling h2 v0.3.12 Compiling tokio-native-tls v0.3.0 Compiling thiserror-impl v1.0.30 Compiling darling_macro v0.10.2 Compiling thiserror v1.0.30 Compiling zip v0.5.13 Compiling darling v0.10.2 Compiling derive_builder_core v0.9.0 Compiling zip-extensions v0.6.1 Compiling rayon-cond v0.1.0 Compiling hyper v0.14.17 Compiling serde_urlencoded v0.7.1 Compiling spm_precompiled v0.1.3 Compiling hyper-tls v0.5.0 Compiling reqwest v0.11.10 Compiling cached-path v0.5.3 Compiling tokenizers v0.11.3 Compiling tokenizers-ruby v0.1.0 (/home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0) warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:63:101 | 63 | fn tokenizers_from_pretrained(identifier: RString, revision: RString, auth_token: AnyObject) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = note: `#[warn(improper_ctypes_definitions)]` on by default = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:88:52 | 88 | fn bpe_new(vocab: RString, merges: RString) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:107:43 | 107 | fn tokenizer_new(model: AnyObject) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:137:43 | 137 | fn tokenizer_encode(text: RString) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `rutie::RString`, which is not FFI-safe --> src/lib.rs:147:40 | 147 | fn tokenizer_decode(ids: Array) -> RString { | ^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:159:53 | 159 | fn tokenizer_decoder_set(decoder: AnyObject) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:167:65 | 167 | fn tokenizer_pre_tokenizer_set(pre_tokenizer: AnyObject) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:175:59 | 175 | fn tokenizer_normalizer_set(normalizer: AnyObject) -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `rutie::Array`, which is not FFI-safe --> src/lib.rs:188:26 | 188 | fn encoding_ids() -> Array { | ^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `rutie::Array`, which is not FFI-safe --> src/lib.rs:198:29 | 198 | fn encoding_tokens() -> Array { | ^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:213:29 | 213 | fn bpe_decoder_new() -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:225:36 | 225 | fn bert_pre_tokenizer_new() -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `extern` fn uses type `AnyObject`, which is not FFI-safe --> src/lib.rs:237:33 | 237 | fn bert_normalizer_new() -> AnyObject { | ^^^^^^^^^ not FFI-safe | = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct = note: this struct has unspecified layout warning: `tokenizers-ruby` (lib) generated 13 warnings Finished release [optimized] target(s) in 1m 51s mv target/release/libtokenizers.so lib/tokenizers/ext.so mv: cannot stat 'target/release/libtokenizers.so': No such file or directory make: *** [Makefile:3: install] Error 1 make failed, exit code 2 Gem files will remain installed in /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0 for inspection. Results logged to /home/ur5us/.rvm/gems/ruby-3.0.4/extensions/x86_64-linux/3.0.0/tokenizers-0.1.0/gem_make.out
Hey @ur5us, thanks for reporting! Just pushed 0.1.1, which fixes the error and warnings.
Getting the following compiler output: