Added vendored build support

jean-airoldie commented 5 years ago

This is the reopening of #262 since github does not allow the reopening of PRs after you force push.

The vendored crate now compiles statically, adding dynamic compilation down the line would be trivial.

hansieodendaal commented 5 years ago

Hi there, I get this error when trying to compile on Windows, using Visual Studio 2017 Developer Command Prompt v15.0:

C:\Code\zmq\rust-zmq-jean-airoldie>cargo build --features vendored
   Compiling zmq-sys v0.9.0 (C:\Code\zmq\rust-zmq-jean-airoldie\zmq-sys)
error: could not find native static library `zmq`, perhaps an -L flag is missing?

error: aborting due to previous error

error: Could not compile `zmq-sys`.

What is the instructions for compiling on Windows?

jean-airoldie commented 5 years ago

I have no experience with compiling on windows and I don't have acces to a windows build but i'll try to help.

The error means that this line is not linking to the library properly, which means that lib_dir is the not the right directory. I think this might be related to this line.

After building for cmake, the binaires will be located in a lib directory. However, the name of this directory changes depending on the os.

The easiest way to test would be to clone zeromq-src then run cargo test --all. Then tell me what the directories looks like.

Here is an example from my end:

$ ls target/debug/build/testcrate-5cb3af760001bbb1/out
`build/  include/  lib64/  share/`
$ ls target/debug/build/testcrate-5cb3af760001bbb1/out/lib64/
libzmq.a  pkgconfig/

In my case, the lib directory would be lib64 since it contains libzmq.a, the static library we want to link against.

jean-airoldie commented 5 years ago

I'll cook try to cook up a appveyor script to try to build on windows. edit: This is my WIP PR.

jean-airoldie commented 5 years ago

Yep running on appveyor I get the same error:

error: could not find native static library `zmq`, perhaps an -L flag is missing?

jean-airoldie commented 5 years ago

Seems like the compiled static lib is called libzmq-v141-mt-sgd-4_3_2.lib but cargo expects zmq.lib. You got any idea why?

hansieodendaal commented 5 years ago

Jean,

Yes, I could also replicate the same behavior on my local system with the appveyer script. The folder structure is shown below, with important files shown. The dll files must be accessible via path. The twolibzmq-v141-mt-sgd-4_3_1.lib files listed are exactly the same.

C:\Code\zmq\rust-zmq-jean-airoldie\target\debug\build\zmq-sys-eaf48a98745f7ff2>tree /f
Folder PATH listing for volume Windows
Volume serial number is 04F5-2962
C:.
│   ...
│
└───out
    ├───bin
    │       concrt140.dll
    │       msvcp140.dll
    │       vcruntime140.dll
    │
    ├───build
    │   │   ...
    │   │
    │   ├───bin
    │   │   └───Debug
    │   │       │   ...
    │   │
    │   ├───CMakeFiles
    │   │   │   ...
    │   │
    │   ├───doc
    │   ├───lib
    │   │   └───Debug
    │   │           libzmq-v141-mt-sgd-4_3_1.lib
    │   │           unity.lib
    │   │
    │   ├───libzmq-static.dir
    │   │   └───Debug
    │   │   │   ...
    │   │
    │   ├───tests
    │   │   │   ...
    │   │
    │   ├───unittests
    │   │   │   ...
    │   │
    │   └───x64
    │       └───Debug
    │           │   ...
    │
    ├───CMake
    │       ...
    │
    ├───include
    │       zmq.h
    │       zmq_utils.h
    │
    ├───lib
    │   │   libzmq-v141-mt-sgd-4_3_1.lib
    │   │
    │   └───pkgconfig
    │           ...
    │
    └───share
        └───zmq
                ...

Running cargo test --all produced this output:

error: failed to run custom build command for `zmq-sys v0.9.0 (C:\Code\zmq\rust-zmq-jean-airoldie\zmq-sys)`
process didn't exit successfully: `C:\Code\zmq\rust-zmq-jean-airoldie\target\debug\build\zmq-sys-56d4f05976e51776\build-script-main` (exit code: 101)
--- stderr
thread 'main' panicked at 'Unable to locate libzmq:
Failed to run `"pkg-config" "--libs" "--cflags" "libzmq >= 4.1"`: The system cannot find the file specified. (os error 2)', zmq-sys\build\pkg_config.rs:25:17
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

jean-airoldie commented 5 years ago

I meant running cargo test --all -vv from zeromq-src.

I'm not sure I understand what you meant by accessible via PATH. I assume that outputting this to cargo should be enought. On unix this would add the directory to the rustc search path.

edit: Added -vv flag so we see the build script in action.

hansieodendaal commented 5 years ago

Yes, I did that from your branch. The lib files reference the dll files in the dynamic case, the latter to be accessible via path. See comments in your WIP PR.

oblique commented 5 years ago

Tested cross-compilation for ARM Linux and works.

jean-airoldie commented 5 years ago

I'll bump the version.

jean-airoldie commented 5 years ago

I'll add CI testing for the feature for both release and debug since it changes how libzmq is compiled.

jean-airoldie commented 5 years ago

Note that until #268 is merged, the CI will fail.

edit: Or not...

hansieodendaal commented 5 years ago

I ran these tests in Windows on this latest branch in my own environment (not Appveyer style) using pre-built release version libzmq libraries for the dynamic linking:

This pass:

cargo clean
cargo test
cargo test --release

This pass:

cargo clean
cargo test --all-features
cargo test --release --all-features

This fails (do not know if the sequence is valid though):

cargo clean
cargo build --features vendored
cargo test
cargo test --release

build test 1.txt

Rust environment:

C:\Users\pluto\Documents\Code\zmq\rust-zmq-jean-airoldie>rustc --version
rustc 1.36.0-nightly (af98304b9 2019-05-11)

C:\Users\pluto\Documents\Code\zmq\rust-zmq-jean-airoldie>cargo --version
cargo 1.36.0-nightly (759b6161a 2019-05-06)

These instructions apply for the dynamic linking case as per PR #249:

Note for Windows users (re. dynamic linking of ZeroMQ):

When building libzmq from sources, the library must be renamed to zmq.lib from the auto named libzmq-v***-mt-gd-*_*_*.lib, libzmq.lib, libzmq-mt-*_*_*.lib, etc.
The folder containing the *.dll (dynamic link library) referred to by zmq.lib must be accessible via the path for the session that invokes the Rust compiler.
The name of the *.dll in question depends on the build system used for libzmq and can usually be seen when opening zmq.lib in a text editor.

jean-airoldie commented 5 years ago

From my understanding the failing tests are due to #268.

jean-airoldie commented 5 years ago

Specifically:

cargo clean
cargo test --release

should fail even on the master branch currently because some tests assume that target/debug exists.

hansieodendaal commented 5 years ago

After applying patch code from PR #268 to this branch these also work:

cargo clean
cargo test --release

cargo clean
cargo test --release --all-features

cargo clean
cargo test --release --features vendored

build test 2.txt

hansieodendaal commented 5 years ago

I also ran these tests in Windows on this latest branch Appveyer style on my own system, but with patches applied from PR #268, with an updated appveyer.bat to enable running it locally (appveyor_local.bat), including the test sequences below. All tests passed.

cargo clean
cargo test

cargo clean
cargo test --release

cargo clean
cargo test --all-features 

cargo clean
cargo test --release --all-features 

cargo clean
cargo test --features vendored

cargo clean
cargo test --release --features vendored

build test 3.txt

jean-airoldie commented 5 years ago

I'll add a NEWS entry.

rotty commented 5 years ago

Thanks a lot!

erickt / rust-zmq

Added vendored build support #267