search

ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby
Apache License 2.0
132 stars 6 forks source link

Error compiling #2

Closed ur5us closed 2 years ago

ur5us commented 2 years ago

Getting the following compiler output:

rustc --version
rustc 1.61.0 (fe5b13d68 2022-05-18)

gem install tokenizers
Building native extensions. This could take a while...
ERROR:  Error installing tokenizers:
    ERROR: Failed to build gem native extension.

    current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers
/home/ur5us/.rvm/rubies/ruby-3.0.4/bin/ruby -I /home/ur5us/.rvm/rubies/ruby-3.0.4/lib/ruby/site_ruby/3.0.0 -r ./siteconf20220623-143041-cdpust.rb extconf.rb

current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers
make DESTDIR\= clean
cargo clean

current directory: /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0/ext/tokenizers
make DESTDIR\=
cargo build --release
   Compiling libc v0.2.121
   Compiling cfg-if v1.0.0
   Compiling autocfg v1.1.0
   Compiling cc v1.0.73
   Compiling pkg-config v0.3.24
   Compiling proc-macro2 v1.0.36
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.89
   Compiling memchr v2.3.4
   Compiling lazy_static v1.4.0
   Compiling log v0.4.14
   Compiling version_check v0.9.4
   Compiling pin-project-lite v0.2.8
   Compiling bitflags v1.3.2
   Compiling futures-core v0.3.21
   Compiling bytes v1.1.0
   Compiling once_cell v1.10.0
   Compiling itoa v1.0.1
   Compiling serde_derive v1.0.136
   Compiling futures-task v0.3.21
   Compiling crossbeam-utils v0.8.8
   Compiling typenum v1.15.0
   Compiling futures-util v0.3.21
   Compiling foreign-types-shared v0.1.1
   Compiling serde v1.0.136
   Compiling fnv v1.0.7
   Compiling openssl v0.10.38
   Compiling ryu v1.0.9
   Compiling pin-utils v0.1.0
   Compiling hashbrown v0.11.2
   Compiling slab v0.4.5
   Compiling crc32fast v1.3.2
   Compiling unicode-width v0.1.9
   Compiling futures-channel v0.3.21
   Compiling native-tls v0.2.8
   Compiling futures-io v0.3.21
   Compiling tinyvec_macros v0.1.0
   Compiling matches v0.1.9
   Compiling httparse v1.6.0
   Compiling futures-sink v0.3.21
   Compiling scopeguard v1.1.0
   Compiling radium v0.5.3
   Compiling percent-encoding v2.1.0
   Compiling strsim v0.9.3
   Compiling ident_case v1.0.1
   Compiling adler v1.0.2
   Compiling rayon-core v1.9.1
   Compiling getrandom v0.1.16
   Compiling ppv-lite86 v0.2.16
   Compiling try-lock v0.2.3
   Compiling regex-syntax v0.6.25
   Compiling openssl-probe v0.1.5
   Compiling httpdate v1.0.2
   Compiling lexical-core v0.7.6
   Compiling unicode-bidi v0.3.7
   Compiling encoding_rs v0.8.30
   Compiling tower-service v0.3.1
   Compiling either v1.6.1
   Compiling byteorder v1.4.3
   Compiling arrayvec v0.5.2
   Compiling tap v1.0.1
   Compiling wyz v0.2.0
   Compiling static_assertions v1.1.0
   Compiling serde_json v1.0.79
   Compiling funty v1.1.0
   Compiling fastrand v1.7.0
   Compiling cpufeatures v0.2.2
   Compiling mime v0.3.16
   Compiling base64 v0.13.0
   Compiling remove_dir_all v0.5.3
   Compiling derive_builder v0.9.0
   Compiling ipnet v2.4.0
   Compiling number_prefix v0.4.0
   Compiling ansi_term v0.12.1
   Compiling glob v0.3.0
   Compiling unicode-segmentation v1.9.0
   Compiling strsim v0.8.0
   Compiling macro_rules_attribute-proc_macro v0.0.2
   Compiling smallvec v1.8.0
   Compiling number_prefix v0.3.0
   Compiling rutie v0.8.3
   Compiling vec_map v0.8.2
   Compiling base64 v0.12.3
   Compiling unicode_categories v0.1.1
   Compiling paste v1.0.6
   Compiling tracing-core v0.1.23
   Compiling indexmap v1.8.0
   Compiling memoffset v0.6.5
   Compiling crossbeam-epoch v0.9.8
   Compiling miniz_oxide v0.4.4
   Compiling rayon v1.5.1
   Compiling generic-array v0.14.5
   Compiling nom v6.2.1
   Compiling foreign-types v0.3.2
   Compiling http v0.2.6
   Compiling textwrap v0.11.0
   Compiling openssl-sys v0.9.72
   Compiling bzip2-sys v0.1.11+1.0.8
   Compiling onig_sys v69.7.1
   Compiling esaxx-rs v0.1.7
   Compiling tinyvec v1.5.1
   Compiling form_urlencoded v1.0.1
   Compiling itertools v0.8.2
   Compiling itertools v0.9.0
   Compiling unicode-normalization-alignments v0.1.12
   Compiling macro_rules_attribute v0.0.2
   Compiling tracing v0.1.32
   Compiling aho-corasick v0.7.15
   Compiling unicode-normalization v0.1.19
   Compiling http-body v0.4.4
   Compiling num_cpus v1.13.1
   Compiling socket2 v0.4.4
   Compiling terminal_size v0.1.17
   Compiling getrandom v0.2.5
   Compiling time v0.1.43
   Compiling filetime v0.2.15
   Compiling xattr v0.2.2
   Compiling atty v0.2.14
   Compiling tempfile v3.3.0
   Compiling dirs-sys v0.3.7
   Compiling fs2 v0.4.3
   Compiling quote v1.0.16
   Compiling mio v0.8.2
   Compiling want v0.3.0
   Compiling crossbeam-channel v0.5.4
   Compiling bitvec v0.19.6
   Compiling regex v1.4.6
   Compiling idna v0.2.3
   Compiling rand_core v0.6.3
   Compiling rand_core v0.5.1
   Compiling tar v0.4.38
   Compiling clap v2.34.0
   Compiling dirs v3.0.2
   Compiling tokio v1.17.0
   Compiling flate2 v1.0.22
   Compiling block-buffer v0.10.2
   Compiling crypto-common v0.1.3
   Compiling rand_chacha v0.3.1
   Compiling rand_chacha v0.2.2
   Compiling url v2.2.2
   Compiling bzip2 v0.4.3
   Compiling console v0.15.0
   Compiling crossbeam-deque v0.8.1
   Compiling digest v0.10.3
   Compiling rand v0.8.5
   Compiling rand v0.7.3
   Compiling tokio-util v0.6.9
   Compiling indicatif v0.16.2
   Compiling indicatif v0.15.0
   Compiling darling_core v0.10.2
   Compiling onig v6.3.1
   Compiling sha2 v0.10.2
   Compiling h2 v0.3.12
   Compiling tokio-native-tls v0.3.0
   Compiling thiserror-impl v1.0.30
   Compiling darling_macro v0.10.2
   Compiling thiserror v1.0.30
   Compiling zip v0.5.13
   Compiling darling v0.10.2
   Compiling derive_builder_core v0.9.0
   Compiling zip-extensions v0.6.1
   Compiling rayon-cond v0.1.0
   Compiling hyper v0.14.17
   Compiling serde_urlencoded v0.7.1
   Compiling spm_precompiled v0.1.3
   Compiling hyper-tls v0.5.0
   Compiling reqwest v0.11.10
   Compiling cached-path v0.5.3
   Compiling tokenizers v0.11.3
   Compiling tokenizers-ruby v0.1.0 (/home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0)
warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
  --> src/lib.rs:63:101
   |
63 |     fn tokenizers_from_pretrained(identifier: RString, revision: RString, auth_token: AnyObject) -> AnyObject {
   |                                                                                                     ^^^^^^^^^ not FFI-safe
   |
   = note: `#[warn(improper_ctypes_definitions)]` on by default
   = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
   = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
  --> src/lib.rs:88:52
   |
88 |     fn bpe_new(vocab: RString, merges: RString) -> AnyObject {
   |                                                    ^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
   = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:107:43
    |
107 |     fn tokenizer_new(model: AnyObject) -> AnyObject {
    |                                           ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:137:43
    |
137 |     fn tokenizer_encode(text: RString) -> AnyObject {
    |                                           ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `rutie::RString`, which is not FFI-safe
   --> src/lib.rs:147:40
    |
147 |     fn tokenizer_decode(ids: Array) -> RString {
    |                                        ^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:159:53
    |
159 |     fn tokenizer_decoder_set(decoder: AnyObject) -> AnyObject {
    |                                                     ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:167:65
    |
167 |     fn tokenizer_pre_tokenizer_set(pre_tokenizer: AnyObject) -> AnyObject {
    |                                                                 ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:175:59
    |
175 |     fn tokenizer_normalizer_set(normalizer: AnyObject) -> AnyObject {
    |                                                           ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `rutie::Array`, which is not FFI-safe
   --> src/lib.rs:188:26
    |
188 |     fn encoding_ids() -> Array {
    |                          ^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `rutie::Array`, which is not FFI-safe
   --> src/lib.rs:198:29
    |
198 |     fn encoding_tokens() -> Array {
    |                             ^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:213:29
    |
213 |     fn bpe_decoder_new() -> AnyObject {
    |                             ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:225:36
    |
225 |     fn bert_pre_tokenizer_new() -> AnyObject {
    |                                    ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `extern` fn uses type `AnyObject`, which is not FFI-safe
   --> src/lib.rs:237:33
    |
237 |     fn bert_normalizer_new() -> AnyObject {
    |                                 ^^^^^^^^^ not FFI-safe
    |
    = help: consider adding a `#[repr(C)]` or `#[repr(transparent)]` attribute to this struct
    = note: this struct has unspecified layout

warning: `tokenizers-ruby` (lib) generated 13 warnings
    Finished release [optimized] target(s) in 1m 51s
mv target/release/libtokenizers.so lib/tokenizers/ext.so
mv: cannot stat 'target/release/libtokenizers.so': No such file or directory
make: *** [Makefile:3: install] Error 1

make failed, exit code 2

Gem files will remain installed in /home/ur5us/.rvm/gems/ruby-3.0.4/gems/tokenizers-0.1.0 for inspection.
Results logged to /home/ur5us/.rvm/gems/ruby-3.0.4/extensions/x86_64-linux/3.0.0/tokenizers-0.1.0/gem_make.out
ankane commented 2 years ago

Hey @ur5us, thanks for reporting! Just pushed 0.1.1, which fixes the error and warnings.