search

archenroot / gentoo-overlay

Gentoo overlay with main focus on GPU, Neural Networks, Big Data and Java technologies
http://gentoo.archenroot.org
15 stars 4 forks source link

tensorflow rebuild - 4 hours and counting, 1 CPU % 100%, no disk activity #34

Open LGTrader opened 6 years ago

LGTrader commented 6 years ago

As far as I can tell there is one python job spinning one CPU core at 100%. My system was switched by Gentoo to python 3.5 today. Is that a problem for the tensorflow ebuild?

It's been a long time since I built tensorflow so I don't know what's reasonable but emerge -e @world is only about 3 hours so this seems excessive.

Did it crash?

LGTrader commented 6 years ago

After running for about 5 hours and no apparent disk activity I shut the machine down for the evening, brought it back up this morning and am trying again. Sadly, same results. About 3.5 hours in so far, no evidence of much disk activity (iotop -o doesn't indicate much is going on) and top says 1 CPU is using close to 100%. The process is owned by user portage and the command is python.

LGTrader commented 6 years ago

Continuing the investigation, I don't know what the task #7513 thinks it's doing but it's been spinning now for nearly 5 hours and iotop says it's running python configure.py.

Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
7513 idle portage 0.00 B/s 0.00 B/s 0.00 % 0.00 % python configure.py

LGTrader commented 6 years ago

Killed the process. Will look for feedback or other ideas.

archenroot commented 6 years ago

@LGTrader I will test on my system, actually your quite lucky because tensor flow is getting attention of Gentoo devs, we will stabilize it and get troug QA and it will be soon part of Gentoo standard tree.

Anyway I will investigate this your issue. I am going to rebuild tensorflow now (I build on 17 core processor (it is Xeon ES version, I had to disable first core as some cpu broadcast issue... so otherwise it is 18 core :-) )

thanks for report.

LGTrader commented 6 years ago

Thanks for looking into this.

I'm on an i7 980X here with 24GB.

Note: Since I reported this problem I took tensorflow off the machine and did a Gentoo complete rebuild using the 17 level profiles and PIE gcc:6.4.0 compiler stuff. Not sure if your machine is using that, or whether it makes a difference. That total rebuild of the machine was about 8 hours for a bit less than 1200 packages.

On Tue, Dec 5, 2017 at 2:25 PM, archenroot notifications@github.com wrote:

@LGTrader https://github.com/lgtrader I will test on my system, actually your quite lucky because tensor flow is getting attention of Gentoo devs, we will stabilize it and get troug QA and it will be soon part of Gentoo standard tree.

Anyway I will investigate this your issue. I am going to rebuild tensorflow now (I build on 17 core processor (it is Xeon ES version, I had to disable first core as some cpu broadcast issue... so otherwise it is 18 core :-) )

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-349446457, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRhXYNZSkBRELr1DcTKZxhDhsa6lBks5s9bSwgaJpZM4Qv4a5 .

archenroot commented 6 years ago

I just wanted to notice the upgrade, as it has consequences....

There are 2 things: New profile (and I also upgraded today) makes gcc 6 default.

Consequence: Nvidia cuda toolkit 8 cannot run on gcc6 (actually I have some workaround, but lets put it this way is not supported). So we need to move to cuda 9.

Now we have another dependency CUDNN and we have now 2 versions: CUDNN 7 for CUDA 8 and CUDNN for CUDA 9 :-))

I was thinking to support both(by some local flags cuda8 and cuda9 :-) ), but as soon as new profile enforces CUDA 9 anyway, I will probably focus simply on CUDA 9 version and start forgetting CUDA 8.

This is little bit problem because not all frameworks I work on support CUDA 9 (some build process config) and much bigger number doesn't support yeat CUDNN 7 from AI frameworks.

Still, we will be working on AI frameworks in general in future to bring them into main branch in Gentoo.

archenroot commented 6 years ago

Regarding this full rebuild I know, I also face this, as due to some dirty things I do, I sometimes rather rebuild completely...

I work now on first stage: captureing from users anonymous system config: /etc/portage content eix-installed -a list of all packages emerge --info

Based on this I will later decide to work on what I call open cluster, will not talk here about details, but when you would like to make this there will be infrastructure which:

Main task is to deliver gcc, qt, chrome, those big packages purely in binary, but letting the user all USE variables to set.

LGTrader commented 6 years ago

Hi, First, thanks for the heads-up on cuda-toolkit and gcc6. I'm emerging nvidia-cuda-toolkit-9.0.176 now to ensure no problems.

Second, this is an overlay so you are free to make any choice you think appropriate but with my limited visibility I cannot see that you should try very hard to support gcc5 and cuda-toolkit-8 anymore.

No rush on any of this for me. tensorflow is just a curiosity in my life but I'm happy to test the ebuilds and report any problems.

Cheers, Mark

On Tue, Dec 5, 2017 at 2:37 PM, archenroot notifications@github.com wrote:

I just wanted to notice the upgrade, as it has consequences....

There are 2 things: New profile (and I also upgraded today) makes gcc 6 default.

Consequence: Nvidia cuda toolkit 8 cannot run on gcc6 (actually I have some workaround, but lets put it this way is not supported). So we need to move to cuda 9.

Now we have another dependency CUDNN and we have now 2 versions: CUDNN 7 for CUDA 8 and CUDNN for CUDA 9 :-))

I was thinking to support both(by some local flags cuda8 and cuda9 :-) ), but as soon as new profile enforces CUDA 9 anyway, I will probably focus simply on CUDA 9 version and start forgetting CUDA 8.

This is little bit problem because not all frameworks I work on support CUDA 9 (some build process config) and much bigger number doesn't support yeat CUDNN 7 from AI frameworks.

Still, we will be working on AI frameworks in general in future to bring them into main branch in Gentoo.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-349449647, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRlizq1hmWznb-hf_qQ9pIPmHQ62aks5s9beLgaJpZM4Qv4a5 .

archenroot commented 6 years ago

I have now migrated fully CUDNN package from this overlay to CUDA 9 only, no more CUDA 8 support and going to try merge this TensorFlow again. will ping with result.

LGTrader commented 6 years ago

Yesterday as a test I built nvidia-cuda-toolkit-9 which went fine.

I tried nvidia-cuda-sdk-9 which failed. If you want info let me know.

I was going to try nvidia-cuda-cudnn but it looked like we needed a newer ebuild which is what I think you've just done? I don't see anything new in portage but will watch.

Let me know if & when you want me to try things out.

Cheers, Mark

On Thu, Dec 7, 2017 at 9:43 AM, archenroot notifications@github.com wrote:

I have now migrated fully CUDNN package from this overlay to CUDA 9 only, no more CUDA 8 support and going to try merge this TensorFlow again. will ping with result.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-350024748, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRiMV9pVRmd5FD91h7aBiuvDU9w0Fks5s-BWZgaJpZM4Qv4a5 .

archenroot commented 6 years ago

compiling locally in the moment...

archenroot commented 6 years ago

Ok, finished compilation with CUDA 9 and CUDNN 7 - latest commit I did today. I am going to run some GPU tensor test and see if all is ok, I see some issue:

scanelf: rpath_security_checks(): Security problem with relative DT_RUNPATH '../local_config_cuda/cuda/lib64:../local_config_cuda/cuda/extras/CUPTI/lib64' in /var/tmp/portage/sci-libs/tensorflow-9999/image/usr/lib64/python2.7/site-packages/tensorflow/libtensorflow_framework.so
scanelf: rpath_security_checks(): Security problem with relative DT_RUNPATH '../local_config_cuda/cuda/extras/CUPTI/lib64' in /var/tmp/portage/sci-libs/tensorflow-9999/image/usr/lib64/python2.7/site-packages/tensorflow/libtensorflow_framework.so
scanelf: rpath_security_checks(): Security problem with relative DT_RUNPATH '../local_config_cuda/cuda/lib64:../local_config_cuda/cuda/extras/CUPTI/lib64' in /var/tmp/portage/sci-libs/tensorflow-9999/image/usr/lib64/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
scanelf: rpath_security_checks(): Security problem with relative DT_RUNPATH '../local_config_cuda/cuda/extras/CUPTI/lib64' in /var/tmp/portage/sci-libs/tensorflow-9999/image/usr/lib64/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so

Message above is more QA related, but lets see.

archenroot commented 6 years ago

Ok, output si promising:

zangetsu@mobile-server ~/devel/proj/ai/tensorflow-examples/examples/1_Introduction $ python helloworld.py 
2017-12-08 02:09:14.972402: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_UNKNOWN
2017-12-08 02:09:14.972452: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: mobile-server
2017-12-08 02:09:14.972463: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: mobile-server
2017-12-08 02:09:14.972502: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 387.34.0
2017-12-08 02:09:14.972788: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  387.34  Tue Nov 21 03:09:00 PST 2017
GCC version:  gcc version 6.4.0 (Gentoo 6.4.0 p1.1) 
"""
2017-12-08 02:09:14.972812: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 387.34.0
2017-12-08 02:09:14.972818: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 387.34.0
Hello, TensorFlow!

I will commit & push and will work on other requirements regarding virutal/jdk and Java API build support via USE flag

archenroot commented 6 years ago

It works with cuda flag on my machine, so I will close this issue.

But important note: On my new machine with 34 threads I had to limit MAKEOPTS to 8 only instead of full 34. The problem was the memory. I have now only temporary 8GB while waiting for next week delivery of 64GB ECC.

The compilation phase was in same situations really slow, but all because of SWAP. Even when I limit to -j11 option, in some cases I saw about 8GB RAM (full) + 17GB SWAP occupied. And these phases looks like "I am not doing anything", so please check this situation if it didn't occur in your case.

LGTrader commented 6 years ago

OK, that's all good news.

I did an eix-sync and tensorflow is asking for ~amd64 nvidia-drivers-387.34. I suspect that means I'm seeing your new ebuild. I'll have to do some testing of that nvidia-driver. I use Matlab on this system and it's sometimes fussy about newish nvidia-drivers. I'll try to make sure that's working first.

You know, I didn't look at whether swapping was the problem. My settings:

EMERGE_DEFAULT_OPTS="--with-bdeps=y --jobs=5" MAKEOPTS="-j13 -l8" PORTAGE_NICENESS=5 PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

suggest that could have been the problem. I would have thought I'd see a lot of disk activity if that happened, and I didn't, but maybe it was so jammed up it wasn't even talking to the drives? I don't know and it really doesn't matter at this point, but I'll keep an eye on that when I build.

It will likely be tomorrow before I report back.

Thanks!

On Thu, Dec 7, 2017 at 11:22 AM, archenroot notifications@github.com wrote:

Closed #34 via 0e41e2f.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

archenroot commented 6 years ago

@LGTrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

LGTrader commented 6 years ago

24GB here so I'll need to cut back.

No need to downgrade the driver at this time. Thanks

On Thu, Dec 7, 2017 at 12:00 PM, archenroot notifications@github.com wrote:

@LGTrader https://github.com/lgtrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-350062217, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRqziRCgWLVGt_32R5zyCVnKppcquks5s-DXLgaJpZM4Qv4a5 .

LGTrader commented 6 years ago

I'm going to be out tonight so I thought I'd get set up to build this. I've updated nvidia-drivers and have no obvious problems with Matlab so that's good.

cudnn is another issue though:

c2RAID6 ~ # emerge -pvDuN nvidia-cuda-cudnn

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild N F ~] dev-libs/nvidia-cuda-cudnn-7.0::archenroot 340643 KiB

Total: 1 package (1 new), Size of downloads: 340643 KiB Fetch Restriction: 1 package (1 unsatisfied)

Fetch instructions for dev-libs/nvidia-cuda-cudnn-7.0:

c2RAID6 ~ # ls -al /usr/portage/distfiles/cudnn-9.0-linux-x64-v7.tgz -rw-r--r-- 1 portage portage 348832287 Dec 7 13:02 /usr/portage/distfiles/cudnn-9.0-linux-x64-v7 .tgz

c2RAID6 ~ #

Is it possible they've updated the package since you downloaded? My file size is somewhat larger than emerge is telling me.

I downloaded the file yesterday and moved it to /usr/portage/distfiles this morning.

Thanks, Mark

On Thu, Dec 7, 2017 at 12:13 PM, Mark Knecht markknecht@gmail.com wrote:

24GB here so I'll need to cut back.

No need to downgrade the driver at this time. Thanks

On Thu, Dec 7, 2017 at 12:00 PM, archenroot notifications@github.com wrote:

@LGTrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

LGTrader commented 6 years ago

Looking at the manifest my file size is slightly different.

Looking at the NVidia Developer's website the newest version is 7.0.5 Dec. 5, 2017 so I suspect for now that's the difference?

On Thu, Dec 7, 2017 at 1:13 PM, Mark Knecht markknecht@gmail.com wrote:

I'm going to be out tonight so I thought I'd get set up to build this. I've updated nvidia-drivers and have no obvious problems with Matlab so that's good.

cudnn is another issue though:

c2RAID6 ~ # emerge -pvDuN nvidia-cuda-cudnn

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild N F ~] dev-libs/nvidia-cuda-cudnn-7.0::archenroot 340643 KiB

Total: 1 package (1 new), Size of downloads: 340643 KiB Fetch Restriction: 1 package (1 unsatisfied)

Fetch instructions for dev-libs/nvidia-cuda-cudnn-7.0:

c2RAID6 ~ # ls -al /usr/portage/distfiles/cudnn-9.0-linux-x64-v7.tgz -rw-r--r-- 1 portage portage 348832287 Dec 7 13:02 /usr/portage/distfiles/cudnn-9.0-linux-x64-v7 .tgz c2RAID6 ~ #

Is it possible they've updated the package since you downloaded? My file size is somewhat larger than emerge is telling me.

I downloaded the file yesterday and moved it to /usr/portage/distfiles this morning.

Thanks, Mark

On Thu, Dec 7, 2017 at 12:13 PM, Mark Knecht markknecht@gmail.com wrote:

24GB here so I'll need to cut back.

No need to downgrade the driver at this time. Thanks

On Thu, Dec 7, 2017 at 12:00 PM, archenroot notifications@github.com wrote:

@LGTrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

LGTrader commented 6 years ago

Very sorry. It was me that downloaded the wrong version from NVidia. cudnn building now.

On Thu, Dec 7, 2017 at 1:23 PM, Mark Knecht markknecht@gmail.com wrote:

Looking at the manifest my file size is slightly different.

Looking at the NVidia Developer's website the newest version is 7.0.5 Dec. 5, 2017 so I suspect for now that's the difference?

  • Mark

On Thu, Dec 7, 2017 at 1:13 PM, Mark Knecht markknecht@gmail.com wrote:

I'm going to be out tonight so I thought I'd get set up to build this. I've updated nvidia-drivers and have no obvious problems with Matlab so that's good.

cudnn is another issue though:

c2RAID6 ~ # emerge -pvDuN nvidia-cuda-cudnn

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild N F ~] dev-libs/nvidia-cuda-cudnn-7.0::archenroot 340643 KiB

Total: 1 package (1 new), Size of downloads: 340643 KiB Fetch Restriction: 1 package (1 unsatisfied)

Fetch instructions for dev-libs/nvidia-cuda-cudnn-7.0:

c2RAID6 ~ # ls -al /usr/portage/distfiles/cudnn-9.0-linux-x64-v7.tgz -rw-r--r-- 1 portage portage 348832287 Dec 7 13:02 /usr/portage/distfiles/cudnn-9.0-linux-x64-v7 .tgz c2RAID6 ~ #

Is it possible they've updated the package since you downloaded? My file size is somewhat larger than emerge is telling me.

I downloaded the file yesterday and moved it to /usr/portage/distfiles this morning.

Thanks, Mark

On Thu, Dec 7, 2017 at 12:13 PM, Mark Knecht markknecht@gmail.com wrote:

24GB here so I'll need to cut back.

No need to downgrade the driver at this time. Thanks

On Thu, Dec 7, 2017 at 12:00 PM, archenroot notifications@github.com wrote:

@LGTrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

LGTrader commented 6 years ago

OK, so the good news is the ebuild ran very fast and didn't use much memory. The downside - I don't think it built anything.

Maybe we need a python3_5 target? My system doesn't have python3.4 on it anymore, as per the Gentoo Python page here:

https://wiki.gentoo.org/wiki/Python

mark@c2RAID6 ~ $ eselect python list Available Python interpreters, in order of preference: [1] python2.7 [2] python3.5 [3] python3.4 mark@c2RAID6 ~ $

My current make.conf python settings are:

PYTHON_TARGETS="python2_7 python3_5" PYTHON_SINGLE_TARGET="python3_5"

As shown below, I can load numpy in both python 2.7 & 3.5 but cannot load tensorflow in either:

mark@c2RAID6 ~ $ python3.5 Python 3.5.4 (default, Dec 6 2017, 16:17:11) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import numpy as np import tensorflow as tf Traceback (most recent call last): File "", line 1, in ImportError: No module named 'tensorflow' exit() mark@c2RAID6 ~ $ python Python 2.7.14 (default, Dec 6 2017, 16:19:02) [GCC 6.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. import numpy as np import tensorflow as tf Traceback (most recent call last): File "", line 1, in ImportError: No module named tensorflow

c2RAID6 ~ # emerge tensorflow Calculating dependencies... done!

Verifying ebuild manifests Emerging (1 of 1) sci-libs/tensorflow-9999::archenroot Installing (1 of 1) sci-libs/tensorflow-9999::archenroot Recording sci-libs/tensorflow in "world" favorites file... Jobs: 1 of 1 complete Load avg: 2.73, 2.64, 2.18

No outdated packages were found on your system.

c2RAID6 ~ # emerge -pv numpy

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild R ~] dev-python/numpy-1.13.3::gentoo USE="lapack -doc {-test}" PYTHON_TARGETS="pytho n2_7 python3_5 -python3_4 -python3_6" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB c2RAID6 ~ # emerge -pvDuN tensorflow

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild N ~] sci-libs/tensorflow-9999::archenroot USE="cuda -opencl" PYTHON_TARGETS="python2 _7 -pypy -python3_4" 0 KiB

Total: 1 package (1 new), Size of downloads: 0 KiB c2RAID6 ~ #

On Thu, Dec 7, 2017 at 1:29 PM, Mark Knecht markknecht@gmail.com wrote:

Very sorry. It was me that downloaded the wrong version from NVidia. cudnn building now.

  • Mark

On Thu, Dec 7, 2017 at 1:23 PM, Mark Knecht markknecht@gmail.com wrote:

Looking at the manifest my file size is slightly different.

Looking at the NVidia Developer's website the newest version is 7.0.5 Dec. 5, 2017 so I suspect for now that's the difference?

  • Mark

On Thu, Dec 7, 2017 at 1:13 PM, Mark Knecht markknecht@gmail.com wrote:

I'm going to be out tonight so I thought I'd get set up to build this. I've updated nvidia-drivers and have no obvious problems with Matlab so that's good.

cudnn is another issue though:

c2RAID6 ~ # emerge -pvDuN nvidia-cuda-cudnn

These are the packages that would be merged, in order:

Calculating dependencies... done! [ebuild N F ~] dev-libs/nvidia-cuda-cudnn-7.0::archenroot 340643 KiB

Total: 1 package (1 new), Size of downloads: 340643 KiB Fetch Restriction: 1 package (1 unsatisfied)

Fetch instructions for dev-libs/nvidia-cuda-cudnn-7.0:

c2RAID6 ~ # ls -al /usr/portage/distfiles/cudnn-9.0-linux-x64-v7.tgz -rw-r--r-- 1 portage portage 348832287 Dec 7 13:02 /usr/portage/distfiles/cudnn-9.0-linux-x64-v7 .tgz c2RAID6 ~ #

Is it possible they've updated the package since you downloaded? My file size is somewhat larger than emerge is telling me.

I downloaded the file yesterday and moved it to /usr/portage/distfiles this morning.

Thanks, Mark

On Thu, Dec 7, 2017 at 12:13 PM, Mark Knecht markknecht@gmail.com wrote:

24GB here so I'll need to cut back.

No need to downgrade the driver at this time. Thanks

On Thu, Dec 7, 2017 at 12:00 PM, archenroot notifications@github.com wrote:

@LGTrader - ok, let me know if driver is an issue, I updated all versions to latest, but can downgrade the driver itself. I understand it can open pandora boxes to update driver :)

How much ram you have? with 13 jobs configured the build will require in total in some moments around ~30-40GB

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

archenroot commented 6 years ago

Looking at the manifest my file size is slightly different.

Looking at the NVidia Developer's website the newest version is 7.0.5 Dec. 5, 2017 so I suspect for now that's the difference?

Yes, this is related to that fact I mentioned as cudnn7 is available for both cuda 8 and cuda 9. I made things as enforced by gcc 6 as default compiler where only cuda 9 is supported (new profile 17), so all tends to cuda 9 only.

Regarding the python I build using python 2.7, but I am going to switch to 3.5 and will compile again...:

mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # eselect python list
Available Python interpreters, in order of preference:
  [1]   python2.7
  [2]   python3.5
  [3]   python3.4
mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # eselect python set 2
mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.5
  [2]   python2.7
  [3]   python3.4
mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # python
Python 3.5.4 (default, Dec  1 2017, 03:01:41) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

I read in detail your output and looking for other differences.... thanks for reporting issues.

My situation is slightly different I switched my python profile back to 2.7 which was situation during tensorflow compilation and now I can do following:

mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # python 
Python 3.5.4 (default, Dec  1 2017, 03:01:41) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import tensorflow as tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'tensorflow'
>>> 
KeyboardInterrupt
>>> 
[3]+  Stopped                 python

mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.5
  [2]   python2.7
  [3]   python3.4
mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # eselect python set 2
mobile-server /opt/cuda/sdk/1_Utilities/deviceQuery # python
Python 2.7.14 (default, Dec  6 2017, 17:32:25) 
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import tensorflow as tf
>>>

So 2.7 works, but none bindings exists for python 3.5

archenroot commented 6 years ago

It is strange that in your case it doesn't work none of those.

archenroot commented 6 years ago

Here is output of: equery f tensorflow (lists all package files) for comparision equery_f_tensorflow-output.txt It was build with python 2.7 set via profile, so looks same as yours... so I wonder why yours doesn't work

here is my output of make.conf:

PYTHON_TARGETS="python2_7 python3_5"
PYTHON_SINGLE_TARGET="python3_5"

Anyway I think this is then new issue to support both targets of python, not just single. > #36

Also for confirmation I use this package: https://github.com/aymericdamien/TensorFlow-Examples

But of course import tensorflow as tf is in general the most basic test the installation works, but maybe you can find it useful.

I am going to switch now profile to python 3.5 and rebuild again and will test:

mobile-server /tmp # python
Python 2.7.14 (default, Dec  6 2017, 17:32:25) 
[GCC 6.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
KeyboardInterrupt
>>> 
[5]+  Stopped                 python
mobile-server /tmp # eselect python list
Available Python interpreters, in order of preference:
  [1]   python2.7
  [2]   python3.5
  [3]   python3.4
mobile-server /tmp # eselect python set 2
mobile-server /tmp # python
Python 3.5.4 (default, Dec  1 2017, 03:01:41) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
archenroot commented 6 years ago

Also I found following definition in ebuild: PYTHON_COMPAT=( python{2_7,3_4} pypy )

That is probably the issue. I also noticed that on Nov 2 was released the 1.4.0 version, where my 9999 version was limited to commit from 2-4 october.

I now create new version 1.4.0 to be the one you should be using and 9999 will be purely only using latest commit (so based on upstream it might or not work)

I will update you later today as I am working as well on some manifest and more use flags to be either enabled or disabled, which can speed up in some cases compilation process (there is hadoop, google cloud, etc. which might not be needed).

Additionally I work on supporting multilanguage binding for C, Go and Java

LGTrader commented 6 years ago

I compiled python3.4 yesterday but it didn't change anything. numpy loads, tensorflow doesn't.

I'll check out the examples you mentioned but importing tensorflow seems a minimum requirement to me.

I'm glad you're doing the 1.4.0 revision. One problem I found others having with other packages was sometimes permissions didn't allow some folks building 9999 packages to actually get the files installed.

I'll continue to watch your work and update as I see changes in portage.

Thanks, Mark

On Fri, Dec 8, 2017 at 5:13 AM, archenroot notifications@github.com wrote:

Also I found following definition in ebuild: PYTHON_COMPAT=( python{2_7,3_4} pypy )

That is probably the issue. I also noticed that on Nov 2 was released the 1.4.0 version, where my 9999 version was limited to commit from 2-4 october.

I now create new version 1.4.0 to be the one you should be using and 9999 will be purely only using latest commit (so based on upstream it might or not work)

I will update you later today as I am working as well on some manifest and more use flags to be either enabled or disabled, which can speed up in some cases compilation process (there is hadoop, google cloud, etc. which might not be needed).

Additionally I work on supporting multilanguage binding for C, Go and Java

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-350249325, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRnkHVuJ2mG5qyUK718s2bxD1CSZXks5s-SfogaJpZM4Qv4a5 .

LGTrader commented 6 years ago

Hi, Not sure exactly where we left this and I'm going to be away for a few days late in the week so please consider this just a status report.

QUESTION: Using FEATURES="keepwork" leaves everything in /var/tmp/portage around but I cannot figure out how to tell what work it did or what problems it had. Can we modify the ebuild to put in some messages?

Cheers, Mark

1) All tensorflow-9999 dependencies are installed explicitly. No problems there.

2) After cleaning up old packages and dist files there's very little tensorflow stuff on the system:

c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

3) Strangely, attempting to fetch tensorflow doesn't actually fetch anything new as best I can tell:

c2RAID6 ~ # emerge -f tensorflow Calculating dependencies... done!

Fetching (1 of 1) sci-libs/tensorflow-9999::archenroot c2RAID6 ~ # updatedb c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

4) Interestingly, actually trying an emerge does download new stuff so maybe fetch just doesn't work? Still, nothing is installed anywhere.

c2RAID6 ~ # emerge tensorflow Calculating dependencies... done!

Verifying ebuild manifests Emerging (1 of 1) sci-libs/tensorflow-9999::archenroot Installing (1 of 1) sci-libs/tensorflow-9999::archenroot Recording sci-libs/tensorflow in "world" favorites file... Jobs: 1 of 1 complete Load avg: 2.19, 1.71, 1.55

No outdated packages were found on your system.

LGTrader commented 6 years ago

OK, using the tensorflow website I got tensorflow sort of working here, at least well enough to run their little example code:

import tensorflow as tf hello = tf.constant('Hello, TensorFlow!') sess = tf.Session() print(sess.run(hello))

Basically, through the work you've done most of it was there, other than tensorflow and virtualenv. I installed virtualenv using portage, then built and activated the private environment.

virtualenv --system-site-packages -p python3 ~/tensorflow source ~/tensorflow/bin/activate easy_install -U pip pip3 install --upgrade tensorflow python

Note that I have python3.5 set as default. The code above runs:

(tensorflow) mark@c2RAID6 ~ $ python Python 3.5.4 (default, Dec 6 2017, 16:17:11) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf hello=tf.constant("Hello, Tensorflow!") sess=tf.Session() print(sess.run(hello)) b'Hello, Tensorflow!' exit() (tensorflow) mark@c2RAID6 ~ $ deactivate mark@c2RAID6 ~ $

Sadly this method doesn't seem to talk to jupyter notebook. It starts the notebook but when I try to execute code in the notebook it doesn't know about tensorflow. maybe I need to start Chrome from inside this env? I don't know.

Anyway, it's a start.

Mark

On Sun, Dec 10, 2017 at 1:17 PM, Mark Knecht markknecht@gmail.com wrote:

Hi, Not sure exactly where we left this and I'm going to be away for a few days late in the week so please consider this just a status report.

QUESTION: Using FEATURES="keepwork" leaves everything in /var/tmp/portage around but I cannot figure out how to tell what work it did or what problems it had. Can we modify the ebuild to put in some messages?

Cheers, Mark

1) All tensorflow-9999 dependencies are installed explicitly. No problems there.

2) After cleaning up old packages and dist files there's very little tensorflow stuff on the system:

c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

3) Strangely, attempting to fetch tensorflow doesn't actually fetch anything new as best I can tell:

c2RAID6 ~ # emerge -f tensorflow Calculating dependencies... done!

Fetching (1 of 1) sci-libs/tensorflow-9999::archenroot c2RAID6 ~ # updatedb c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

4) Interestingly, actually trying an emerge does download new stuff so maybe fetch just doesn't work? Still, nothing is installed anywhere.

c2RAID6 ~ # emerge tensorflow Calculating dependencies... done!

Verifying ebuild manifests Emerging (1 of 1) sci-libs/tensorflow-9999::archenroot Installing (1 of 1) sci-libs/tensorflow-9999::archenroot Recording sci-libs/tensorflow in "world" favorites file... Jobs: 1 of 1 complete Load avg: 2.19, 1.71, 1.55

  • Messages for package sci-libs/tensorflow-9999:

  • Compile Phase - Bazel configured

  • Compile Phase - Bazel build finished

    Auto-cleaning packages...

No outdated packages were found on your system.

  • GNU info directory index is up-to-date. c2RAID6 ~ # updatedb c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/FETCH_HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/config /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/description /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/info /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/applypatch-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/commit-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/post-update.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-applypatch.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-commit.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-push.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-rebase.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-receive.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/prepare-commit-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/update.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/info/exclude /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/info /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack/pack-29907788edc39ad 93ea190a2c9ba5426c1a33e49.idx /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack/pack-29907788edc39ad 93ea190a2c9ba5426c1a33e49.pack /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/heads /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /main /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /old /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.5.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.6.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.10.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.10.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.12.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.7.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.7.1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.8.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.9.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.9.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-alpha /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.3.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.3.0-rc1 /usr/portage/packages/sci-libs/tensorflow-9999.tbz2 /var/db/pkg/sci-libs/tensorflow-9999 /var/db/pkg/sci-libs/tensorflow-9999/BINPKGMD5 /var/db/pkg/sci-libs/tensorflow-9999/BUILD_TIME /var/db/pkg/sci-libs/tensorflow-9999/CATEGORY /var/db/pkg/sci-libs/tensorflow-9999/CBUILD /var/db/pkg/sci-libs/tensorflow-9999/CFLAGS /var/db/pkg/sci-libs/tensorflow-9999/CHOST /var/db/pkg/sci-libs/tensorflow-9999/CONTENTS /var/db/pkg/sci-libs/tensorflow-9999/COUNTER /var/db/pkg/sci-libs/tensorflow-9999/CXXFLAGS /var/db/pkg/sci-libs/tensorflow-9999/DEFINED_PHASES /var/db/pkg/sci-libs/tensorflow-9999/DEPEND /var/db/pkg/sci-libs/tensorflow-9999/DESCRIPTION /var/db/pkg/sci-libs/tensorflow-9999/EAPI /var/db/pkg/sci-libs/tensorflow-9999/FEATURES /var/db/pkg/sci-libs/tensorflow-9999/HOMEPAGE /var/db/pkg/sci-libs/tensorflow-9999/INHERITED /var/db/pkg/sci-libs/tensorflow-9999/IUSE /var/db/pkg/sci-libs/tensorflow-9999/IUSE_EFFECTIVE /var/db/pkg/sci-libs/tensorflow-9999/KEYWORDS /var/db/pkg/sci-libs/tensorflow-9999/LDFLAGS /var/db/pkg/sci-libs/tensorflow-9999/LICENSE /var/db/pkg/sci-libs/tensorflow-9999/PF /var/db/pkg/sci-libs/tensorflow-9999/PKGUSE /var/db/pkg/sci-libs/tensorflow-9999/RDEPEND /var/db/pkg/sci-libs/tensorflow-9999/RESTRICT /var/db/pkg/sci-libs/tensorflow-9999/SIZE /var/db/pkg/sci-libs/tensorflow-9999/SLOT /var/db/pkg/sci-libs/tensorflow-9999/USE /var/db/pkg/sci-libs/tensorflow-9999/environment.bz2 /var/db/pkg/sci-libs/tensorflow-9999/repository /var/db/pkg/sci-libs/tensorflow-9999/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #
LGTrader commented 6 years ago

With jupyter installed in the virtual environment jupyter notebook works perfectly.

I don't have much python experience - I'm doing this work to break away from Matlab over time and to use tensorflow more - but I think the virtual environment is potentially the way to go. Most everything is installed with pip. It's installed in the environment so it doesn't conflict with anything portage installed and portage doesn't even know about it. It does take up disk space in my /home/mark directory but it's easy to backup which I appreciate. I don't know yet why - for a single user machine like mine - where a virtual environment wouldn't be preferred. For an enterprise setup it probably doesn't make as much sense but that's not me.

On Sun, Dec 10, 2017 at 3:36 PM, Mark Knecht markknecht@gmail.com wrote:

OK, using the tensorflow website I got tensorflow sort of working here, at least well enough to run their little example code:

import tensorflow as tf hello = tf.constant('Hello, TensorFlow!') sess = tf.Session() print(sess.run(hello))

Basically, through the work you've done most of it was there, other than tensorflow and virtualenv. I installed virtualenv using portage, then built and activated the private environment.

virtualenv --system-site-packages -p python3 ~/tensorflow source ~/tensorflow/bin/activate easy_install -U pip pip3 install --upgrade tensorflow python

Note that I have python3.5 set as default. The code above runs:

(tensorflow) mark@c2RAID6 ~ $ python Python 3.5.4 (default, Dec 6 2017, 16:17:11) [GCC 6.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf hello=tf.constant("Hello, Tensorflow!") sess=tf.Session() print(sess.run(hello)) b'Hello, Tensorflow!' exit() (tensorflow) mark@c2RAID6 ~ $ deactivate mark@c2RAID6 ~ $

Sadly this method doesn't seem to talk to jupyter notebook. It starts the notebook but when I try to execute code in the notebook it doesn't know about tensorflow. maybe I need to start Chrome from inside this env? I don't know.

Anyway, it's a start.

Mark

On Sun, Dec 10, 2017 at 1:17 PM, Mark Knecht markknecht@gmail.com wrote:

Hi, Not sure exactly where we left this and I'm going to be away for a few days late in the week so please consider this just a status report.

QUESTION: Using FEATURES="keepwork" leaves everything in /var/tmp/portage around but I cannot figure out how to tell what work it did or what problems it had. Can we modify the ebuild to put in some messages?

Cheers, Mark

1) All tensorflow-9999 dependencies are installed explicitly. No problems there.

2) After cleaning up old packages and dist files there's very little tensorflow stuff on the system:

c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

3) Strangely, attempting to fetch tensorflow doesn't actually fetch anything new as best I can tell:

c2RAID6 ~ # emerge -f tensorflow Calculating dependencies... done!

Fetching (1 of 1) sci-libs/tensorflow-9999::archenroot c2RAID6 ~ # updatedb c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #

4) Interestingly, actually trying an emerge does download new stuff so maybe fetch just doesn't work? Still, nothing is installed anywhere.

c2RAID6 ~ # emerge tensorflow Calculating dependencies... done!

Verifying ebuild manifests Emerging (1 of 1) sci-libs/tensorflow-9999::archenroot Installing (1 of 1) sci-libs/tensorflow-9999::archenroot Recording sci-libs/tensorflow in "world" favorites file... Jobs: 1 of 1 complete Load avg: 2.19, 1.71, 1.55

  • Messages for package sci-libs/tensorflow-9999:

  • Compile Phase - Bazel configured

  • Compile Phase - Bazel build finished

    Auto-cleaning packages...

No outdated packages were found on your system.

  • GNU info directory index is up-to-date. c2RAID6 ~ # updatedb c2RAID6 ~ # locate tensorflow /home/mark/CODE/TensorFlow-Examples/examples/5_DataManagement/tensorflow_dataset_api.py /home/mark/CODE/TensorFlow-Examples/notebooks/5_DataManagement/tensorflow_dataset_api.ipynb /home/mark/CODE/handson-ml/09_up_and_running_with_tensorflow.ipynb /home/mark/CODE/handson-ml/12_distributed_tensorflow.ipynb /home/mark/CODE/handson-ml/images/tensorflow /home/mark/CODE/handson-ml/images/tensorflow/README /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/FETCH_HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/config /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/description /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/info /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/applypatch-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/commit-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/post-update.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-applypatch.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-commit.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-push.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-rebase.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/pre-receive.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/prepare-commit-msg.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/hooks/update.sample /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/info/exclude /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/info /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack/pack-29907788edc39ad 93ea190a2c9ba5426c1a33e49.idx /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/objects/pack/pack-29907788edc39ad 93ea190a2c9ba5426c1a33e49.pack /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/heads /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/HEAD /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /main /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/git-r3/sci-libs/tensorflow/0 /old /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.12.1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.5.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/0.6.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.10.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.10.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.11.0rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.12.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.7.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.7.1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.8.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.9.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v0.9.0rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-alpha /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.0.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.1.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc1 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.2.0-rc2 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.3.0-rc0 /usr/portage/distfiles/git3-src/tensorflow_tensorflow.git/refs/tags/v1.3.0-rc1 /usr/portage/packages/sci-libs/tensorflow-9999.tbz2 /var/db/pkg/sci-libs/tensorflow-9999 /var/db/pkg/sci-libs/tensorflow-9999/BINPKGMD5 /var/db/pkg/sci-libs/tensorflow-9999/BUILD_TIME /var/db/pkg/sci-libs/tensorflow-9999/CATEGORY /var/db/pkg/sci-libs/tensorflow-9999/CBUILD /var/db/pkg/sci-libs/tensorflow-9999/CFLAGS /var/db/pkg/sci-libs/tensorflow-9999/CHOST /var/db/pkg/sci-libs/tensorflow-9999/CONTENTS /var/db/pkg/sci-libs/tensorflow-9999/COUNTER /var/db/pkg/sci-libs/tensorflow-9999/CXXFLAGS /var/db/pkg/sci-libs/tensorflow-9999/DEFINED_PHASES /var/db/pkg/sci-libs/tensorflow-9999/DEPEND /var/db/pkg/sci-libs/tensorflow-9999/DESCRIPTION /var/db/pkg/sci-libs/tensorflow-9999/EAPI /var/db/pkg/sci-libs/tensorflow-9999/FEATURES /var/db/pkg/sci-libs/tensorflow-9999/HOMEPAGE /var/db/pkg/sci-libs/tensorflow-9999/INHERITED /var/db/pkg/sci-libs/tensorflow-9999/IUSE /var/db/pkg/sci-libs/tensorflow-9999/IUSE_EFFECTIVE /var/db/pkg/sci-libs/tensorflow-9999/KEYWORDS /var/db/pkg/sci-libs/tensorflow-9999/LDFLAGS /var/db/pkg/sci-libs/tensorflow-9999/LICENSE /var/db/pkg/sci-libs/tensorflow-9999/PF /var/db/pkg/sci-libs/tensorflow-9999/PKGUSE /var/db/pkg/sci-libs/tensorflow-9999/RDEPEND /var/db/pkg/sci-libs/tensorflow-9999/RESTRICT /var/db/pkg/sci-libs/tensorflow-9999/SIZE /var/db/pkg/sci-libs/tensorflow-9999/SLOT /var/db/pkg/sci-libs/tensorflow-9999/USE /var/db/pkg/sci-libs/tensorflow-9999/environment.bz2 /var/db/pkg/sci-libs/tensorflow-9999/repository /var/db/pkg/sci-libs/tensorflow-9999/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow /var/lib/layman/archenroot/sci-libs/tensorflow/Manifest /var/lib/layman/archenroot/sci-libs/tensorflow/files /var/lib/layman/archenroot/sci-libs/tensorflow/metadata.xml /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.5.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-0.6.0.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/tensorflow-9999.ebuild /var/lib/layman/archenroot/sci-libs/tensorflow/files/0.5.0-adjust-configure.patch c2RAID6 ~ #
archenroot commented 6 years ago

@LGTrader - Hi buddy, I don't have time for this in the moment. Working on some AI network to work over about 30 crypto currency exchanges, cool stuff. so will come back to this during Spring I think.

archenroot commented 6 years ago

But thanks for the all input, I have some colaborator for Tensorflow and will ask at science project if someone might be interested as well, its too much work for one person(me) who works on multiple projects hahaha

LGTrader commented 6 years ago

Not a problem right now for me, and possibly with all the system updates it has gotten better anyway. I'll look at it one of these days when I get some time myself.

Thanks, Mark

On Fri, Feb 9, 2018 at 7:38 AM, archenroot notifications@github.com wrote:

But thanks for the all input, I have some colaborator for Tensorflow and will ask at science project if someone might be interested as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/archenroot/gentoo-overlay/issues/34#issuecomment-364451194, or mute the thread https://github.com/notifications/unsubscribe-auth/AEfnRtWURA49TGc65dHkom_kWrbIapvmks5tTFhfgaJpZM4Qv4a5 .