Open andreybokhanko opened 3 years ago
Hi @andreybokhanko ,
We have noticed this problem and will update the Dockerfile with the proper Python dependencies soon.
In the meanwhile, if you are more interested in using CPF rather than reproducing our results in the Perspective paper, I highly encourage you to follow the instructions here bootstrap and build the most up-to-date CPF and its dependencies from scratch. Then in run make compare.out
under tests/regression/*/src
to generate both the parallelized binary and the sequential binary and run the performance test. The time will stored at parallel.time
and seq.time
respectively. Currently, 2mm
, 3mm
, covariance
, and correlation
should work.
Hi @vgene ,
Thanks for the quick response!
Yes, ultimately I'm interested in actually using CPF -- I wanted to check your Perspective results to verify that they work in my environment / use case. Unfortunately, master branch of cpf also fails in my environment (Ubuntu 20.04, gcc 9.3.0) on building llvm step. I used unmodified Makefile.example file from your bootstrap directory.
Perhaps there is a problem in my environment. Could you, please, provide a Docker file for an environment that successfully builds current CPF master?
Yours, Andrey
For Ubuntu 20.04, which has the newer glibc, a patch is needed to compile LLVM 9, which is included here:
https://github.com/PrincetonUniversity/cpf/blob/master/bootstrap/diff-llvm-on-glibc231.patch
You just need to apply this patch to llvm-project with git apply path/to/diff-llvm-on-glibc231.patch
in llvm-project directory, which should be cloned in to your cpf-workspace in your home directory. After that, rerun make all
in bootstrap
.
Also, when copying Makefile.example to Makefile, make sure you edit this line to point to your cloned path to cpf:
cpf-root-path=/your/local/cpf/root
A Dockerfile for the current environment is a great idea and we will make sure it's available by the end of this week.
A fix for the Perspective paper artifact is available, @gchan510 will post it today
Please use this Dockerfile, it contains updates for Python 3.8.
FROM ubuntu:16.04
RUN apt-get update --fix-missing
RUN apt-get install -y curl
RUN apt-get install -y vim make gcc-5 g++-5 time binutils ruby software-properties-common
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 10
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 10
# Install more up-to-date python
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update
RUN apt-get install -y python3.8 python3.8-dev python3.8-venv
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 10
RUN python3 -m ensurepip
RUN pip3 install --upgrade pip
COPY ./requirements.txt /root/requirements.txt
RUN pip3 install -r /root/requirements.txt
RUN useradd --create-home --shell /bin/bash asplos20ae
USER asplos20ae
WORKDIR /home/asplos20ae
COPY --chown=asplos20ae ./benchmarks /home/asplos20ae/benchmarks
COPY --chown=asplos20ae ./CK /home/asplos20ae/CK
COPY --chown=asplos20ae ./exp_scripts /home/asplos20ae/exp_scripts
COPY --chown=asplos20ae ./llvm /home/asplos20ae/llvm
COPY --chown=asplos20ae ./perspective_lib /home/asplos20ae/perspective_lib
COPY --chown=asplos20ae ./README.md /home/asplos20ae/README.md
COPY --chown=asplos20ae ./reference-result.txt /home/asplos20ae/reference-result.txt
COPY --chown=asplos20ae ./reference-comparison-exp.pdf /home/asplos20ae/reference-comparison-exp.pdf
COPY --chown=asplos20ae ./reference-scalability-exp.pdf /home/asplos20ae/reference-scalability-exp.pdf
COPY --chown=asplos20ae ./bashrc /home/asplos20ae/bashrc
RUN cat /home/asplos20ae/bashrc >> /home/asplos20ae/.bashrc
RUN rm /home/asplos20ae/bashrc
CMD /bin/bash
The requirements.txt
also need a change. Please use this updated one:
joblib>=0.13.2
numpy>=1.16.1
ptyprocess>=0.6.0
termcolor>=1.1.0
requests>=2.21.0
GitPython>=2.1.11
ck==1.11.4
pandas>=0.22
psutil
scipy
matplotlib
@vgene , @gchan510 , thank you for providing the fixes in such a short time!
I'm going on a two-weeks vacation; will try your fixes after returning from the vacation in May -- and report results here.
Yours, Andrey
I tried the dockerfile that @gchan510 posted (along with requirements.txt
updates) -- it works fine, thanks!
I suggest to keep this issue open until the updated dockerfile + requirement.txt would be committed and updated instructions for artifacts reproduction be published.
@andreybokhanko Awesome! Also note that there's a dockerfile now in the master branch. You can try to use that to build the latest CPF.
In the meanwhile, if you are more interested in using CPF rather than reproducing our results in the Perspective paper, I highly encourage you to follow the instructions here bootstrap and build the most up-to-date CPF and its dependencies from scratch. Then in run
make compare.out
undertests/regression/*/src
to generate both the parallelized binary and the sequential binary and run the performance test. The time will stored atparallel.time
andseq.time
respectively. Currently,2mm
,3mm
,covariance
, andcorrelation
should work.
Unfortunately, this fails for me for current ToT version (git sha1 is a5d5a4398b6c68478848ff4949fc4e0b6f0be45b).
Build finishes successfully, but executing 2mm
and 3mm
leads to core dump (on benchmark.collaborative.exe
execution step), dijkstra-dynsize
fails to compile (it crashes on noelle step), both covariance
and correlation
fail to link. I used make benchmark.compare.out
command for all of the tests.
Thank you for the feedback! Were you using the Dockerfile? It seems like a setup issue, the latest regression is passing https://github.com/PrincetonUniversity/cpf/runs/2725055202?check_suite_focus=true. We will double check the compilation inside docker.
Could you give a brief summary of the setup including OS version, gcc and glibc version?
Yes, this is from a container using the Dockerfile you supplied -- so this is Ubuntu 20.04, gcc 9.3.0, glibc 2.31, etc.
Hi,
Sorry if this is a wrong place to submit such an issue -- in this case, let me know which one would be more appropriate.
After a lot of trial and error, I discovered that instructions from your "A. Artifact Appendix" section of ASPLO'20 paper (https://liberty.princeton.edu/Publications/asplos20_perspective.pdf) don't work.
Specifically, "docker build" command fails with the following error:
This can be solved by adding "RUN apt-get update" as the first RUN line; however, then we stumble on installation of required python packages:
Apparently, a lot of packages (most crucially, scipy) require Python 3.7 as the minimum, while Ubuntu 16.04 comes with Python 3.5.
I failed to solve this problem with Ubuntu 16.04 and switched to Ubuntu 20.04 (which doesn't have gcc-5, so corresponding lines have to be removed). Dockerfile that works for me is:
Unfortunately, even with this dockerfile the final step, that is supposed to run benchmarking, doesn't work:
I'm not an expert in "ck" framework, so stumbled here.
Please kindly help me to move forward or -- preferably -- post fixed instructions somewhere.
Yours, Andrey \=== Advanced Software Technology Lab Huawei