search

kakao / buffalo

TOROS Buffalo: A fast and scalable production-ready open source project for recommender systems
Apache License 2.0
576 stars 106 forks source link

Installation in Mac environment is very difficult #7

Closed shalomeir closed 5 years ago

shalomeir commented 5 years ago

Is this code for only in LINUX? Very hard to install in MAC OSX environment. I tried many things and tweak a little code to complete installation. But I couldn't succeed. In my Mac, default gcc command is not properly works cause that this repository gcc version is higher than default. So I use gcc-9 that I install by brew and execute with prefix CC="gcc-9". And install command is not complete when building 'cbuffalo' extension step.

In another way, can someone share a Dockerfile plz?

ummae commented 5 years ago

@shalomeir Okay, we didn't test buffalo on Mac. IMHO we could but, compiling on Mac has some problems to solve. e.g. dependency on gnu sort. I will provide Docker file first.

ummae commented 5 years ago

@shalomeir I just added Dockerfile on ./examples directory. Please check it out and tell me is it works to you. (I tested it on MacOS Mojave 10.14.6)

shalomeir commented 5 years ago

@shalomeir I just added Dockerfile on ./examples directory. Please check it out and tell me is it works to you. (I tested it on MacOS Mojave 10.14.6)

@ummae Thanks a lot. I'll check it in a day or two.

shalomeir commented 5 years ago

@ummae I gave up because it was too difficult to run. Dockerfile created an image well. And I execute 'python ./example_als.py example1 in examples directory'. Also I download 'ml-100k' files and change some file names such as main and iid. But it's not works. So I followed just basic starting point guided at documentation 'nosetests ./tests/algo/test_algo.py -v'. Also it it not works.

ImportError: cannot import name 'log'

Overall, it is not kind and seems too difficult to reproduce.

ummae commented 5 years ago

@shalomeir How did you run the docker container? Would you let me know the steps to reproduce the behavior?

shalomeir commented 5 years ago

@ummae I did as it was written in this file. https://github.com/kakao/buffalo/blob/dev/examples/README.md and '''bash cd ./buffalo.git nosetests ./tests/algo/test_als.py -v '''

Also I also download movie lenz file to my local tests/ext folder and sharing by -v command.

ummae commented 5 years ago

@shalomeir Did you run preprocessing.py? https://github.com/kakao/buffalo/tree/dev/tests#preprcessing

and what is output log when you type nosetests?

shalomeir commented 5 years ago

@ummae It doesn't work. Did you do the same procedure using Dockerfile too? There are some miss matched file name or directorie name between code and repository that is downloaded by Dockerfile. I think that container based on same Dockerfile could be almost same whenever I run my local mac or not.

ummae commented 5 years ago

@shalomeir Yes, it works. I tested on two Mac OS Machines, Mac Mini, MacBook Pro. Let me know all commands and its output messages, then I can examine what was problem more easily.

examples $> docker build -t buffalo.dev .
examples $> docker run -ti -e LC_ALL=C.UTF-8 buffalo.dev /bin/bash
root@98ff7e95906e ~# cd /home/toros && source ./venv/bin/activate && cd buffalo.git/tests/ext
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# wget http://files.grouplens.org/datasets/movielens/ml-100k.zip
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# wget http://files.grouplens.org/datasets/movielens/ml-20m.zip
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# wget http://mattmahoney.net/dc/text8.zip
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# unzip ml-100k.zip
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# unzip ml-20m.zip
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests/ext# mkdir text8; mv text8.zip; cd text8;unzip text8.zip; cd ../../
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests# python preprocess.py
[INFO    ] 2019-09-02 12:01:32 [preprocess.py:104] preprocessing for matrix market format of ml-20m...
[INFO    ] 2019-09-02 12:02:46 [preprocess.py:134] preprocessing for stream format of ml-20m...
[WARNING ] 2019-09-02 12:03:03 [preprocess.py:160] Cannot find the brunch directory
Traceback (most recent call last):
  File "preprocess.py", line 199, in <module>
    prepare_dataset()
  File "preprocess.py", line 194, in prepare_dataset
    make_mm_from_stream('./ext/brunch/', './ext/brunch/mm')
  File "preprocess.py", line 24, in make_mm_from_stream
    uids = {u.strip(): idx + 1 for idx, u in enumerate(open(user_path))}
FileNotFoundError: [Errno 2] No such file or directory: './ext/brunch/uid'
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests# cd ..
(venv) root@98ff7e95906e:/home/toros/buffalo.git/# nosetests --nologcapture --nocapture -v ./algo/test_als.py
(venv) root@98ff7e95906e:/home/toros/buffalo.git/tests# nosetests --nologcapture --nocapture -v ./algo/test_als.py
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/toros/venv/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
[ERROR   ] 2019-09-02 12:04:24 [als.py:22] ImportError CuALS, no cuda library exists. error message: No module named 'buffalo.algo.cuda'
test0_get_default_option (algo.test_als.TestALS) ... ok
test10_fast_most_similar (algo.test_als.TestALS) ... ok
test11_train_ml_20m_on_gpu (algo.test_als.TestALS) ... ERROR
test1_is_valid_option (algo.test_als.TestALS) ... ok
test2_init_with_dict (algo.test_als.TestALS) ... [INFO    ] 2019-09-02 12:04:28 [als.py:55] ALS({
  "evaluation_on_learning": true,
  "compute_loss_on_training": true,
  "early_stopping_rounds": 0,
  "save_best": false,
  "evaluation_period": 1,
  "save_period": 10,
  "random_seed": 0,
  "validation": {},
  "adaptive_reg": false,
  "save_factors": false,
  "accelerator": false,
  "d": 20,
  "num_iters": 10,
  "num_workers": 1,
  "hyper_threads": 256,
  "num_cg_max_iters": 3,
  "reg_u": 0.1,
  "reg_i": 0.1,
  "alpha": 8,
  "optimizer": "manual_cg",
...

Since kakao-brunch-12m, kakao-reco-730m databases are not required for unittest, we can ignore error from preprocessing about brunch dataset. Anyway let me know the error message from you console.

shalomeir commented 5 years ago

Thank you for describing the whole process. It didn't go a bit, but it worked out well.

First, error at this step

unzip ml-100k.zip unzip: command not found

So I install unzip

2nd error at this step

mkdir text8; mv text8.zip; cd text8;unzip text8.zip; cd ../../

As you might expect, it wasn't a big deal

3rd error at this step

nosetests --nologcapture --nocapture -v ./algo/test_als.py bash: nosetests: command not found

So I install nose and try again. Also should be go upper directory /tests And.. Finally it worked out.

It's the same as easy steps, but of course there are some things that I can't understand when I encounter some error whenever I followed by the official document. I hope it would be a more accurate guide in the official document.

Anyway thank you for your patience.

ummae commented 5 years ago

@shalomeir Congrats! Okay, I will. Enjoy then.