search

ponylang / corral

:horse: Pony dependency manager tool
https://ponylang.io
BSD 2-Clause "Simplified" License
194 stars 20 forks source link

Handle "native" dependencies #142

Open damon-kwok opened 4 years ago

damon-kwok commented 4 years ago

This was edited by @SeanTAllen:

This started as an idea to use vcpkg from @damon-kwok.

https://github.com/Microsoft/vcpkg Microsoft's vcpkg provides cross-platform c/c++ library installation. It works well with pony ffi.

It has since expanded into a conversation for handling native dependencies.

damon-kwok commented 4 years ago

I think corral's support for vcpkg only needs to include its lib file directory in the search path, so that users do not need to fill in the full path.

SeanTAllen commented 4 years ago

Corral doesn't have anything to do with c/c++ library installation. Rather than have an issue for this, I think it is important to first decide if and how we want to approach non-pony dependencies with corral.

@kulibali have you given any thought to non-pony dependencies and corral?

chalcolith commented 4 years ago

I haven't played with vcpkg any. I'm a bit hesitant to introduce an external dependency to corral. My tentative plan, which I swear I'm going to get around to any time now, is to introduce the ability to download binaries and put them in the PONY_PATH. The use case is merely to get the crypto and ssl libraries working on windows so that I can get ponyup working.

SeanTAllen commented 4 years ago

@kulibali any thoughts on how to do in a distro/OS dependent way?

chalcolith commented 4 years ago

The idea would be to have a section in corral.json like "libs": { "windows": { "bin": "foo.dll", "url": "https:foo.zip" }, "linux": {...} } and then corral would uncompress the archive, find the required bin file, and add the directory it's in to PONY_PATH.

damon-kwok commented 4 years ago

There are many compilers and distro/OS, and the burden of precompilation can be heavy. Provide the source address, using CORRAL to call VC GCC clang in local compilation, it can minimize the burden, maximum compatibility. There are many libraries that offer a number of options that users may want to tap into the compilation process.

SeanTAllen commented 4 years ago

@kulibali I think "linux" isn't expansive enough given the desire folks might have to use a package manager etc.

SeanTAllen commented 4 years ago

There are many compilers and distro/OS, and the burden of precompilation can be heavy. Provide the source address, using CORRAL to call VC GCC clang in local compilation, it can minimize the burden, maximum compatibility. There are many libraries that offer a number of options that users may want to tap into the compilation process.

It's unclear to me what you are suggesting.

SeanTAllen commented 4 years ago

Can anyone comment on what Rust's cargo does for this?

damon-kwok commented 4 years ago

Simply put, I don't think Corral needs to provide precompiled files for users to download. We can learn from the treatment of VCPKG. It simply provides the source code address and the script for the compilation process. Even so, this is a lot of work, so at this stage we can choose to use VCPKG directly. When I ran "VCPKG install openssl", I found the experience great and it worked well on non-Windows systems

SeanTAllen commented 4 years ago

What does " Corral needs to provide precompiled files for users to download" mean to you? I don't think anyone has suggested that Corral provides the libraries. Rather it can allow for them to be installed. Is that what you mean?

damon-kwok commented 4 years ago

I saw that Cargo didn't do a good job of that. It just used the library that came with the system. But I have noticed that the Rust ecosystem thrives, for example if you want to use MongoDB, the MongoDB team provides the Rust driver, so FFI is always an alternative to the Rust. So FFI is more important for Pony.

SeanTAllen commented 4 years ago

I saw that Cargo didn't do a good job of that.

Didn't do a good job of what @damon-kwok ?

damon-kwok commented 4 years ago

Corral needs to provide precompiled files for users to download"

I mean there's a lot of work involved in providing pre-compiled binaries.

damon-kwok commented 4 years ago

I saw that Cargo didn't do a good job of that.

Cargo depends on the system library version.

SeanTAllen commented 4 years ago

I saw that Cargo didn't do a good job of that.

Cargo has provided no additional support for this

What is "this"?

damon-kwok commented 4 years ago

Caogo hasn't done anything, but can we do better

SeanTAllen commented 4 years ago

Corral needs to provide precompiled files for users to download"

I mean there's a lot of work involved in providing pre-compiled binaries.

You keep saying "provided". I don't know what "provided" means to you in this context. To me it suggests that you think that we the pony team would compile binaries of libraries and store them somewhere for people to download using corral.

No one has suggested that we provide any precompiled binaries of the sort I mention above. Rather there is a question of "should we make it possible for people to use corral to install precompiled binaries".

SeanTAllen commented 4 years ago

We discussed this during the sync call today (July 7, 2020 if you want to refer to it once uploaded).

General thoughts:

After Theo mentioned how cargo handles (he will be detailing later), we think "that's eh" but we don't have better ideas.

We think that allowing the usage of tools like brew, apt etc to install packages would be very difficult. Many package managers don't allow you to select a version and for a specific release of code that uses a foreign library, that would be problematic. Supporting installation via package manager seems very unlikely.

Allowing for corral in some fashion to download and build libraries from source and then install them in a pony specific fashion however, seems doable and something we want to look into more as a first step.

What that means as a first step needs to be determined further.

damon-kwok commented 4 years ago

Yes, I may have misunderstood the sentence:

The idea would be to have a section in corral.json like "libs": { "windows": { "bin": "foo.dll", "url": "https:foo.zip" }, "linux": {...} } and then corral would uncompress the archive, find the required bin file, and add the directory it's in to PONY_PATH.

damon-kwok commented 4 years ago

Allowing for corral in some fashion to download and build libraries from source and then install them in a pony specific fashion however, seems doable and something we want to look into more as a first step.

This is the right approach. the first thing we need to do is to determine the format of the description file, which contains the platform-wide compilation process script.

chalcolith commented 4 years ago

It occurs to me that since we already have the pattern of including a Makefile and make.ps1 in library project to do things like testing, we could have Corral call that (perhaps with a special target thirdparty or something) if present, and then the Makefile would get and compile the external libs and put them in the directory that Corral is going to add to PONY_PATH.

SeanTAllen commented 4 years ago

I want to get away from using Makefiles. I'd like to roll all that functionality into corral.