Open sparks-code opened 1 year ago
Hey @sparks-code sorry for the slow response.
Bumping to LLVM 16 will require code changes. The pass list will be a little different, and there may be some other API changes. To get you started, you could:
Unfortunately there's no simple recipe for this. It may require some work. Hope that helps.
Cheers, Chris
We have a long term project to overhaul the build system in a way that would make it much easier to plug in different versions of LLVM (including custom forks), but sadly no bandwidth to work on this at the moment.
hello,@ChrisCummins.thanks for your advice. I'm already trying to upgrade the llvm version, buthttps://github.com/facebookresearch/CompilerGym/blob/development/compiler_gym/envs/llvm/service/RunService.ccthis file generat errors during the build process, In the new version of LLVM, some functions are gone. so So how should I change the initLlvm() function?
Hey @sparks-code sorry for the slow response.
Bumping to LLVM 16 will require code changes. The pass list will be a little different, and there may be some other API changes. To get you started, you could:
- Build from source using cmake and an external version of LLVM 10 (as a sanity test).
- Swap out to external LLVM 16 in your cmake config, try and build, iterate on build errors.
- Update the action files in https://github.com/facebookresearch/CompilerGym/tree/development/compiler_gym/envs/llvm/service/passes/10.0.0 to the LLVM 16 pass list, adding any new ones and removing old ones.
Unfortunately there's no simple recipe for this. It may require some work. Hope that helps.
Cheers, Chris
Hi @sparks-code, good to here that you're making progress.
Is the build error happening when you switch the LLVM to 16? Does the build error happen using the default LLVM 10? If not, then it must be some change in LLVM's initialization routines. You could check the changelogs to see how those routines changed. Your best bet for help on LLVM-specific questions will be on their discord, I'm not an LLVM dev.
Cheers, Chris
Hey @sparks-code sorry for the slow response.
Bumping to LLVM 16 will require code changes. The pass list will be a little different, and there may be some other API changes. To get you started, you could:
- Build from source using cmake and an external version of LLVM 10 (as a sanity test).
- Swap out to external LLVM 16 in your cmake config, try and build, iterate on build errors.
- Update the action files in https://github.com/facebookresearch/CompilerGym/tree/development/compiler_gym/envs/llvm/service/passes/10.0.0 to the LLVM 16 pass list, adding any new ones and removing old ones.
Unfortunately there's no simple recipe for this. It may require some work. Hope that helps.
Cheers, Chris
Hi Chris, I followed all these steps. It appears that I can build the whl
package, and reinstall via
pip install <path to build directory>/py_pkg/dist/compiler_gym*.whl --force-reinstall
However, then I tried env.reset
for llvm-v0
. It gives errors. I am now puzzled by this error. do you have any clue on this? thanks.
details here: https://github.com/facebookresearch/CompilerGym/issues/788#issuecomment-1531979923
I tried to dive into it a little bit.. The FileNotFoundError
is not informative.. I tried to use an existing bc
for reset
, still same error, where I am 100% certain the bc
file is there..
>>> import os
>>> os.path.exists("/home/cgao3/qsort.bc")
True
>>> import compiler_gym
>>> env = compiler_gym.make("llvm-v0")
>>> env.compiler_version
'12.0.1 x86_64-unknown-linux-gnu'
>>> b = env.make_benchmark("/home/cgao3/qsort.bc")
>>> env.reset(benchmark=b)
Traceback (most recent call last):
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 760, in _reset
error, reply = _call_with_error(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 695, in _call_with_error
return None, self.service(stub_method, *args, **kwargs)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/connection.py", line 813, in __call__
return self.connection(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/connection.py", line 217, in __call__
raise FileNotFoundError(e.details()) from None
FileNotFoundError: Benchmark not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/connection.py", line 201, in __call__
return stub_method(request, timeout=timeout)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Connection reset by peer"
debug_error_string = "{"created":"@1683309164.001394705","description":"Error received from peer ipv6:[::1]:33199","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Connection reset by peer","grpc_status":14}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 695, in _call_with_error
return None, self.service(stub_method, *args, **kwargs)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/connection.py", line 813, in __call__
return self.connection(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/connection.py", line 230, in __call__
raise ServiceIsClosed("Service is offline")
compiler_gym.errors.service_errors.ServiceIsClosed: Service is offline
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/envs/llvm/llvm_env.py", line 366, in reset
return super().reset(*args, **kwargs)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 624, in reset
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 681, in _retry
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 681, in _retry
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 681, in _retry
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 681, in _retry
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 681, in _retry
return self._reset(
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 778, in _reset
return _retry(error)
File "/home/cgao3/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym/service/client_service_compiler_env.py", line 675, in _retry
raise OSError(
OSError: Failed to reset environment using benchmark benchmark://file-v0/home/cgao3/qsort.bc after 4 attempts.
Last error (ServiceIsClosed): Service is offline
Followed up on #788.
Hi, I'm doing the same job. Have you built successfully so far?
hello,@ChrisCummins.thanks for your advice. I'm already trying to upgrade the llvm version, buthttps://github.com/facebookresearch/CompilerGym/blob/development/compiler_gym/envs/llvm/service/RunService.ccthis file generat errors during the build process, In the new version of LLVM, some functions are gone. so So how should I change the initLlvm() function?
Hey @sparks-code sorry for the slow response. Bumping to LLVM 16 will require code changes. The pass list will be a little different, and there may be some other API changes. To get you started, you could:
- Build from source using cmake and an external version of LLVM 10 (as a sanity test).
- Swap out to external LLVM 16 in your cmake config, try and build, iterate on build errors.
- Update the action files in https://github.com/facebookresearch/CompilerGym/tree/development/compiler_gym/envs/llvm/service/passes/10.0.0 to the LLVM 16 pass list, adding any new ones and removing old ones.
Unfortunately there's no simple recipe for this. It may require some work. Hope that helps. Cheers, Chris
Hi, I'm doing the same job. Have you built successfully so far?
Hi, I'm doing the same job. Have you built successfully so far? I have upgraded, but the pass manager of llvm17 seems to be different from 10, and some passes have changed. If you are interested, you can add me on WeChat 17356055726 and communicate with me.
Hi, I'm doing the same job. Have you built successfully so far? I have upgraded, but the pass manager of llvm17 seems to be different from 10, and some passes have changed. If you are interested, you can add me on WeChat 17356055726 and communicate with me.
Okay, I got it. Thank you.
How to modify the binary downloaded by ~/.local/share/compiler_gym/llvm-v0
, I build compile_gym through llvm11, I manually replace the llvm-0 downloaded by compiler_gym with the binary of llvm-11, the .bc file generated by llvm_wrapper is not available Use (the final generated binary has runtime problems)
❓ Questions and Help
Hello,ChrisCummins! thank you again for your last help! I noticed the compiler_ The default compiler version used by Gym is 10. If I want to update it to 16 and possibly modify the action space, what do I need to do?
Additional Context