search

xmake-io / xmake

🔥 A cross-platform build utility based on Lua
https://xmake.io
Apache License 2.0
10.08k stars 784 forks source link

v3.0 Roadmap #5527

Open waruqi opened 2 months ago

waruqi commented 2 months ago

If there are proposed changes that you would like to see introduced in xmake v3.0, you can submit them here.

It will be adopted if the vote result >= 80%.

we can use set_policy("compatibility.version", "3.0") or xmake f --policies=compatibility.version:3.0 to attempt v3.0

Roadmap (Adopted)

Roadmap (Draft)

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

It can be solved, but in this case, environment variables must be set for each project, which is a bit cumbersome. I have made it clear that xmake does not need to use the tool chain under a specific path.

Does each project use a different toolchain? Generally, one or two global virtual environments will suffice.

Now this is the case, 1 PATH only contains the environment variable of bin in msys2 2 If you use xmake and msvc to compile the project, it will mix the libraries in msys2 (this seems to be the case, anyway, it is a mixture of libraries from different tool chains) , it was mentioned in the issue before, but it can’t be found. In the end, it was normal to remove the msys2 library)

I now hope that xmake will ignore the library in msys2 that I added in the environment variable.

This is very strange. The library name of msys2 is xxx.dll.a, and msvc is .lib. The formats are different and should not be linked, unless tools such as msys2 gfortran are used; in this case, replace your own flang/intel fortran Putting it in front of gfortran can also solve the problem. I feel that the fundamental problem is not in the environment variables. I am looking for the system toolchain/library mechanism in xmake.

heheda123123 commented 1 month ago

可以解决，不过这样的话，每个项目都得设置一遍环境变量，略繁琐，本身我是明确了不需要xmake使用特定路径下的工具链

每个项目都用不同的toolchain吗？一般有一两个全局虚拟环境就可以了吧

现在是这样， 1 PATH里面只有msys2中bin的环境变量 2 如果用xmake使用msvc去编译项目，他会混合使用到msys2中的库(好像是这种情况，反正是混合了不同工具链的库，之前issue提过，但是找不到了，最后就是去掉msys2的库才正常) 我现在是希望xmake不管我在环境变量中加的这个msys2中的库

这个很奇怪啊，msys2的库名字是xxx.dll.a，msvc是.lib，格式都不一样不应该链接上的，除非用了msys2 gfortran之类工具；这种情况把自己的flang/intel fortran放gfortran前面也能解决问题。感觉根本问题不在环境变量，在xmake找系统toolchain/library的机制

找到了，是这个issue https://github.com/xmake-io/xmake/issues/4479

主要问题应该是“path 里面加了 pkg-config”，xmake会使用这个msys下的pkg-config

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

It can be solved, but in this case, each project has to set the environment variables, which is a bit cumbersome. I have made it clear that xmake does not need to use the tool chain under a specific path.

Does each project use a different toolchain? Generally, one or two global virtual environments will suffice.

Now this is the case, 1 PATH only contains the environment variable of bin in msys2 2 If you use xmake and msvc to compile the project, it will mix the libraries in msys2 (this seems to be the case, anyway, it is a mix of different tool chains) The library was mentioned in the previous issue, but it could not be found. In the end, it was normal to remove the msys2 library) I now hope that xmake will ignore the library in msys2 that I added in the environment variable.

This is very strange. The library name of msys2 is xxx.dll.a, and msvc is .lib. The formats are different and should not be linked, unless tools such as msys2 gfortran are used; in this case, change your own flang/intel Putting fortran in front of gfortran can also solve the problem. I feel that the fundamental problem is not in the environment variables. I am looking for the system toolchain/library mechanism in xmake.

Found it, it’s this issue https://github.com/xmake-io/xmake/issues/4479

xq114 commented 1 month ago

Found it, it’s this issue #4479

但 xmake 无法通过 pkg-config 去判断返回的库是 gnulib 还是 mslib 格式，只能假定都可使用。

这个是可以判断的，parse完linkdir之后 https://github.com/xmake-io/xmake/blob/dev/xmake/modules/package/manager/pkgconfig/find_package.lua#L88-L89 这里区分一下后缀名.a/.lib就行了。尽管这里msys2的zlib msvc也能用，但混用mingw格式包的这种行为还是默认禁用为好，实在要用可以加一个policy 控制 是否对msvc启用mingw格式的包

xq114 commented 1 month ago

可以新开一个issue

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

Found it, it’s this issue #4479

But xmake cannot determine whether the returned library is in gnulib or mslib format through pkg-config, and can only assume that both can be used.

This can be judged after parse linkdir https://github.com/xmake-io/xmake/blob/dev/xmake/modules/package/manager/pkgconfig/find_package.lua#L88-L89 Let’s distinguish the suffix names here .a/.lib will do. Although zlib msvc of msys2 can also be used here, the behavior of mixing mingw format packages is better to be disabled by default. If you really want to use it, you can add a policy to control whether to enable mingw format packages for msvc.

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

You can open a new issue

waruqi commented 1 month ago

可以解决，不过这样的话，每个项目都得设置一遍环境变量，略繁琐，本身我是明确了不需要xmake使用特定路径下的工具链

每个项目都用不同的toolchain吗？一般有一两个全局虚拟环境就可以了吧

现在是这样， 1 PATH里面只有msys2中bin的环境变量 2 如果用xmake使用msvc去编译项目，他会混合使用到msys2中的库(好像是这种情况，反正是混合了不同工具链的库，之前issue提过，但是找不到了，最后就是去掉msys2的库才正常) 我现在是希望xmake不管我在环境变量中加的这个msys2中的库

这个很奇怪啊，msys2的库名字是xxx.dll.a，msvc是.lib，格式都不一样不应该链接上的，除非用了msys2 gfortran之类工具；这种情况把自己的flang/intel fortran放gfortran前面也能解决问题。感觉根本问题不在环境变量，在xmake找系统toolchain/library的机制

找到了，是这个issue #4479

主要问题应该是“path 里面加了 pkg-config”，xmake会使用这个msys下的pkg-config

这个到 #4479 里面去讨论，或者新开 issues 就行了。

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

It can be solved, but in this case, environment variables must be set for each project, which is a bit cumbersome. I have made it clear that xmake does not need to use the tool chain under a specific path.

Does each project use a different toolchain? Generally, one or two global virtual environments will suffice.

Now this is the case, 1 PATH only contains the environment variable of bin in msys2 2 If you use xmake and msvc to compile the project, it will mix the libraries in msys2 (this seems to be the case, anyway, it is a mix of different tool chains) The library was mentioned in the previous issue, but it could not be found. In the end, it was normal to remove the msys2 library) I now hope that xmake will ignore the library in msys2 that I added in the environment variable.

This is very strange. The library name of msys2 is xxx.dll.a, and msvc is .lib. The formats are different and should not be linked, unless tools such as msys2 gfortran are used; in this case, change your own flang/ Putting intel fortran in front of gfortran can also solve the problem. I feel that the fundamental problem is not in the environment variables. I am looking for the system toolchain/library mechanism in xmake.

Found it, it’s issue #4479

The main problem should be that "pkg-config is added to the path", xmake will use this pkg-config under msys

Just discuss this in #4479, or open a new issue.

Arteiimis commented 1 month ago

https://github.com/xmake-io/xmake/issues/5609

heheda123123 commented 1 month ago

5609

支持，现在的musl.cc工具链只能从linux上跨平台/架构编译。这个工具链能大大扩展交叉编译的能力

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

5609

Supported, the current musl.cc toolchain can only be compiled cross-platform/architecture from Linux. This tool chain can greatly expand cross-compilation capabilities

waruqi commented 1 month ago

We can use set_policy("compatibility.version", "3.0") or xmake f --policies=compatibility.version:3.0 to attempt v3.0

https://github.com/xmake-io/xmake/commit/f9196dfa7073e3c3511230072d606c89f28a075d

waruqi commented 1 month ago

New Poll: Autobuild when calling xmake run https://github.com/xmake-io/xmake/discussions/5624

waruqi commented 1 month ago

Now we can use set_policy("compatibility.version", "3.0") to switch vs runtime to MD/Md

https://github.com/xmake-io/xmake/pull/5626

SnowinterCat commented 1 month ago

我需要一个接口来遍历我的项目依赖的三方库的列表，我需要将三方库的二进制打包到我的安装包中。 我现在的做法是在脚本域中使用以下代码来获取三方库的二进制的：

        import("core.project.project")
        cprint("${yellow}Export Shared libs")
        for pkg_name, info in pairs(project:required_packages()) do
            local libfiles = info:get("libfiles")
        end

但是这只能获取到三方库的库文件，对于一些拥有插件或者工具的三方库，例如gstreamer、ffmpeg、qt，我需要用到他们的工具和插件，并将其打包到安装包中。目前我没有办法做到直接从xmake的脚本域中找到它们 能否提供一个函数例如project:all_packages(),target:all_packages()来获取到他们的manifest信息？

又或者为package的包管理中添加一个on_export_runtime()，可以用于一键导出三方库的运行时？

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

I need an interface to traverse the list of third-party libraries that my project depends on, and I need to package the binaries of the third-party libraries into my installation package. My current approach is to use the following code in the script domain to obtain the binary of the third-party library:

import("core.project.project")
cprint("${yellow}Export Shared libs")
for pkg_name, info in pairs(project:required_packages()) do
local libfiles = info:get("libfiles")
end

But this can only obtain the library files of third-party libraries. For some third-party libraries with plug-ins or tools, such as gstreamer, ffmpeg, and qt, I need to use their tools and plug-ins and package them into the installation package. Currently I have no way to find them directly from xmake's script domain Can you provide a function such as project:all_packages(), target:all_packages() to obtain their manifest information?

Or add an on_export_runtime() to the package management of the package, which can be used to export the runtime of third-party libraries with one click?

waruqi commented 1 month ago

我需要一个接口来遍历我的项目依赖的三方库的列表，我需要将三方库的二进制打包到我的安装包中。 我现在的做法是在脚本域中使用以下代码来获取三方库的二进制的：

        import("core.project.project")
        cprint("${yellow}Export Shared libs")
        for pkg_name, info in pairs(project:required_packages()) do
            local libfiles = info:get("libfiles")
        end

但是这只能获取到三方库的库文件，对于一些拥有插件或者工具的三方库，例如gstreamer、ffmpeg、qt，我需要用到他们的工具和插件，并将其打包到安装包中。目前我没有办法做到直接从xmake的脚本域中找到它们 能否提供一个函数例如project:all_packages(),target:all_packages()来获取到他们的manifest信息？

又或者为package的包管理中添加一个on_export_runtime()，可以用于一键导出三方库的运行时？

这种问题，没必要在这里问，可以到 discussion 里提，target:pkgs() ，pkg:installdir() 原本就能取到。

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

I need an interface to traverse the list of third-party libraries that my project depends on, and I need to package the binaries of the third-party libraries into my installation package. My current approach is to use the following code in the script domain to obtain the binary of the third-party library:

import("core.project.project")
cprint("${yellow}Export Shared libs")
for pkg_name, info in pairs(project:required_packages()) do
local libfiles = info:get("libfiles")
end

But this can only obtain the library files of third-party libraries. For some third-party libraries with plug-ins or tools, such as gstreamer, ffmpeg, and qt, I need to use their tools and plug-ins and package them into the installation package. Currently, I have no way to find them directly from the script domain of xmake. Can you provide a function such as project:all_packages(), target:all_packages() to get their manifest information?

Or add an on_export_runtime() to the package management of the package, which can be used to export the runtime of third-party libraries with one click?

There is no need to ask this kind of question here. You can mention it in the discussion. Target:pkgs() and pkg:installdir() can be obtained originally.

Wzshun commented 3 weeks ago

A way to change config without having to respecify all config, maybe separate some configs such as mode from other configs. It's quite painful to work with a lot of config because of that.

I also want it, something like a file xmake.lua.user, and xmake build will check it if change, auto config with it.

jingkaimori commented 2 weeks ago

c51和sdcc工具链的默认plat不要设为当前系统，建议为裸机（没有系统的单片机，自立实现）添加名为bare或者standalone的目标平台plat属性值。c51工具链的默认架构arch建议也弄成mcs51，不然下面的日志太难绷：

> xmake config --clean
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.40.33811
> xmake build --rebuild
[ 50%]: cache compiling.release src\main.c
[ 75%]: linking.release hello
[ 75%]: generating.release build\windows\x64\release\hello.hex
[100%]: build ok, spent 0.125s

这个无解，每个 target 可以绑定多个不同 toolchain 的，而全局平台 toolchain 只有一个，跟他们独立。

那xmake-repo里的包，如何适配c51或者sdcc工具链，同时不影响默认工具链的编译？

waruqi commented 2 weeks ago

c51和sdcc工具链的默认plat不要设为当前系统，建议为裸机（没有系统的单片机，自立实现）添加名为bare或者standalone的目标平台plat属性值。c51工具链的默认架构arch建议也弄成mcs51，不然下面的日志太难绷：

> xmake config --clean
checking for platform ... windows
checking for architecture ... x64
checking for Microsoft Visual Studio (x64) version ... 2022
checking for Microsoft C/C++ Compiler (x64) version ... 19.40.33811
> xmake build --rebuild
[ 50%]: cache compiling.release src\main.c
[ 75%]: linking.release hello
[ 75%]: generating.release build\windows\x64\release\hello.hex
[100%]: build ok, spent 0.125s

这个无解，每个 target 可以绑定多个不同 toolchain 的，而全局平台 toolchain 只有一个，跟他们独立。

那xmake-repo里的包，如何适配c51或者sdcc工具链，同时不影响默认工具链的编译？

add_requires("xxx", {configs = {toolchains = ""}}) 也是可以设置 toolchain 的

SirLynix commented 2 weeks ago

How would you feel about having 

add_requires("xx", { configs = { foo = "bar" } })

automatically add

add_requireconfs("**.xx", { configs = { foo = "bar" }})

?

Also 

add_requires("libsdl 2.30")

would add

add_requireconfs("**.libsdl", { version = "2.30" }})

This way, package config and versions would automatically replicate to their dependencies as well, because currently:

add_requires("libsdl 2.28", "libsdl_ttf")

will make xmake install this:

$ xmake f --check
checking for Microsoft Visual Studio (x64) version ... 2022
note: install or modify (m) these packages (pass -y to skip confirm)?
in xmake-repo:
  -> libsdl 2.28.5 [runtimes:"MD"]
  -> libsdl#1 2.30.8 [runtimes:"MD", from:libsdl_ttf]
  -> libsdl_ttf 2.22.0 [runtimes:"MD"]
please input: y (y/n/m)

same goes for

add_requires("libsdl", { configs = {sdlmain = false}})
add_requires("libsdl_ttf")

=>

in xmake-repo:
  -> libsdl 2.30.8 [runtimes:"MD", sdlmain:n]
  -> libsdl#1 2.30.8 [runtimes:"MD", from:libsdl_ttf]
  -> libsdl_ttf 2.22.0 [runtimes:"MD"]
please input: y (y/n/m)

it's been a long time I've been thinking about making a feature request for a policy for this, and I think it would be a great default to have that in xmake 3.0.

what do you think?

(also thinking about it, package.librarydeps.strict_compatibility would be great to enable by default in that case, to avoid subtles bugs when you change your mind about a package config also used as a dependency)

waruqi commented 2 weeks ago

How would you feel about having add_requires("xx", { configs = { foo = "bar" } }) automatically add add_requireconfs("**.xx", { configs = { foo = "bar" }})? Also add_requires("libsdl 2.30") => add_requireconfs("**.libsdl", { version = "2.30" }}).

This way, package config and versions would automatically replicate to their dependencies as well, because currently:

add_requires("libsdl 2.28", "libsdl_ttf")

will make xmake install this:

$ xmake f --check
checking for Microsoft Visual Studio (x64) version ... 2022
note: install or modify (m) these packages (pass -y to skip confirm)?
in xmake-repo:
  -> libsdl 2.28.5 [runtimes:"MD"]
  -> libsdl#1 2.30.8 [runtimes:"MD", from:libsdl_ttf]
  -> libsdl_ttf 2.22.0 [runtimes:"MD"]
please input: y (y/n/m)

same goes for

add_requires("libsdl", { configs = {sdlmain = false}})
add_requires("libsdl_ttf")

=>

in xmake-repo:
  -> libsdl 2.30.8 [runtimes:"MD", sdlmain:n]
  -> libsdl#1 2.30.8 [runtimes:"MD", from:libsdl_ttf]
  -> libsdl_ttf 2.22.0 [runtimes:"MD"]
please input: y (y/n/m)

it's been a long time I've been thinking about making a feature request for a policy for this, and I think it would be a great default to have that in xmake 3.0.

what do you think?

(also thinking about it, package.librarydeps.strict_compatibility would be great to enable by default in that case, to avoid subtles bugs when you change your mind about a package config also used as a dependency)

The add_requires at the user's top level are completely independent. They should not affect the dependency configurations of other packages, as this can lead to many hidden problems.

SirLynix commented 2 weeks ago

The add_requires at the user's top level are completely independent. They should not affect the dependency configurations of other packages, as this can lead to many hidden problems.

But in the examples I'm showing this is already causing many issues, if you decide to use libsdl as a shared library, libsdl_ttf will use it as a static library causing bugs (libsdl uses global variables which would not be the same between libsdl and libsdl_ttf in that case).

Another example would be openssl, if you use package locking for example and you're locking your openssl package version, it won't lock it as a dependency (and openssl is used by many packages).
Your app cannot have two differents versions of openssl and it will error out on init.

Having that behavior as an opt-out seems safer to me

waruqi commented 2 weeks ago

The add_requires at the user's top level are completely independent. They should not affect the dependency configurations of other packages, as this can lead to many hidden problems.

But in the examples I'm showing this is already causing many issues, if you decide to use libsdl as a shared library, libsdl_ttf will use it as a static library causing bugs (libsdl uses global variables which would not be the same between libsdl and libsdl_ttf in that case).

so we should use add_requireconfs

Another example would be openssl, if you use package locking for example and you're locking your openssl package version, it won't lock it as a dependency (and openssl is used by many packages).

This is just another issue about requires-lock. We can improve requires-lock instead of add_requires.

Your app cannot have two differents versions of openssl and it will error out on init.

However, two completely independent targets can be bound to different versions of OpenSSL. If their dependency behaviors are globally modified through add_requires, these unrelated targets will all be affected.

Having that behavior as an opt-out seems safer to me

In the future, I will try to improve and solve this problem, but not by modifying the default behavior of add_requires. Modifying it would lead to even more problems than now.

SirLynix commented 2 weeks ago

I understand even though I don't agree on the current behavior being safer than what I suggest (especially because it causes subtle bugs one can fix only with advanced xmake usage and knowing how libs depends on each other, my students fall in this trap every year).

(About package lock, the issue arise simply when you set a package version through add_requires, at best it duplicates symbols and at worst it causes weird bugs in your app)

How about having this behavior as an opt-in policy? Because even though we can already do this using add_requireconfs, it means duplicating every add_requires with a add_requireconfs which is redundant and more error-prone.

waruqi commented 2 weeks ago

I understand even though I don't agree on the current behavior being safer than what I suggest (especially because it causes subtle bugs one can fix only with advanced xmake usage and knowing how libs depends on each other, my students fall in this trap every year).

(About package lock, the issue arise simply when you set a package version through add_requires, at best it duplicates symbols and at worst it causes weird bugs in your app)

How about having this behavior as an opt-in policy? Because even though we can already do this using add_requireconfs, it means duplicating every add_requires with a add_requireconfs which is redundant and more error-prone.

I haven't figured out how to improve and solve it yet. You can open an issue first.

Bluelari commented 1 week ago

A way to change config without having to respecify all config, maybe separate some configs such as mode from other configs. It's quite painful to work with a lot of config because of that.

这个+1，甚至希望xmake config的默认行为就是只修改而不要清除之前的config，类似于git config。并且如果真的需要覆盖已有config的话也可以先跑xmake config --clean再修改