search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.72k stars 5.73k forks source link

ray.tune.error.AbortTrialExecution: Trial runner reuse requires reset_trial() to be implemented and return True. #5284

Closed zhaokang1228 closed 5 years ago

zhaokang1228 commented 5 years ago

System information

Describe the problem

This code is what I modified /ray-0.7.2/python/ray/tune/examples/tune_mnist_ray_hyperband.py.

<!-- Describe the problem clearly here. -->
**error logs**
2019-07-26 20:28:16,409 INFO pbt.py:82 -- [explore] perturbed config from {'learning_rate': 0.0008309294089614463, 'activation': 'relu'} -> {'learning_rate': 0.000664743527169157, 'activation': 'tanh'}
2019-07-26 20:28:16,410 INFO pbt.py:304 -- [exploit] transferring weights from trial TrainMNIST_6_activation=relu,learning_rate=0.00083093 (score 0.9399999976158142) -> TrainMNIST_0_activation=relu,learning_rate=1.0741e-05 (score 0.25999999046325684)
2019-07-26 20:28:16,414 ERROR ray_trial_executor.py:211 -- Error starting runner for Trial TrainMNIST_0_activation=relu,learning_rate=1.0741e-05@perturbed[activation=tanh,learning_rate=0.00066474]
Traceback (most recent call last):
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 209, in start_trial
    self._start_trial(trial, checkpoint)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 138, in _start_trial
    or trial._checkpoint.value is not None)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 99, in _setup_runner
    "Trial runner reuse requires reset_trial() to be "
ray.tune.error.AbortTrialExecution: Trial runner reuse requires reset_trial() to be implemented and return True.

Source code / logs

code 

#!/usr/bin/env python
#
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""A deep MNIST classifier using convolutional layers.
See extensive documentation at
https://www.tensorflow.org/get_started/mnist/pros
"""
# Disable linter warnings to maintain consistency with tutorial.
# pylint: disable=invalid-name
# pylint: disable=g-bad-import-order

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import argparse
import time

import ray
from ray import tune
from ray.tune import grid_search, Trainable, sample_from
from tensorflow.examples.tutorials.mnist import input_data
from ray.tune.schedulers import PopulationBasedTraining

import tensorflow as tf
import numpy as np

activation_fn = None  # e.g. tf.nn.relu

def setupCNN(x):
    """setupCNN builds the graph for a deep net for classifying digits.
    Args:
        x: an input tensor with the dimensions (N_examples, 784), where 784 is
        the number of pixels in a standard MNIST image.
    Returns:
        A tuple (y, keep_prob). y is a tensor of shape (N_examples, 10), with
        values equal to the logits of classifying the digit into one of 10
        classes (the digits 0-9). keep_prob is a scalar placeholder for the
        probability of dropout.
    """
    # Reshape to use within a convolutional neural net.
    # Last dimension is for "features" - there is only one here, since images
    # are grayscale -- it would be 3 for an RGB image, 4 for RGBA, etc.
    with tf.name_scope("reshape"):
        x_image = tf.reshape(x, [-1, 28, 28, 1])

    # First convolutional layer - maps one grayscale image to 32 feature maps.
    with tf.name_scope("conv1"):
        W_conv1 = weight_variable([5, 5, 1, 32])
        b_conv1 = bias_variable([32])
        h_conv1 = activation_fn(conv2d(x_image, W_conv1) + b_conv1)

    # Pooling layer - downsamples by 2X.
    with tf.name_scope("pool1"):
        h_pool1 = max_pool_2x2(h_conv1)

    # Second convolutional layer -- maps 32 feature maps to 64.
    with tf.name_scope("conv2"):
        W_conv2 = weight_variable([5, 5, 32, 64])
        b_conv2 = bias_variable([64])
        h_conv2 = activation_fn(conv2d(h_pool1, W_conv2) + b_conv2)

    # Second pooling layer.
    with tf.name_scope("pool2"):
        h_pool2 = max_pool_2x2(h_conv2)

    # Fully connected layer 1 -- after 2 round of downsampling, our 28x28 image
    # is down to 7x7x64 feature maps -- maps this to 1024 features.
    with tf.name_scope("fc1"):
        W_fc1 = weight_variable([7 * 7 * 64, 1024])
        b_fc1 = bias_variable([1024])

        h_pool2_flat = tf.reshape(h_pool2, [-1, 7 * 7 * 64])
        h_fc1 = activation_fn(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

    # Dropout - controls the complexity of the model, prevents co-adaptation of
    # features.
    with tf.name_scope("dropout"):
        keep_prob = tf.placeholder(tf.float32)
        h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

    # Map the 1024 features to 10 classes, one for each digit
    with tf.name_scope("fc2"):
        W_fc2 = weight_variable([1024, 10])
        b_fc2 = bias_variable([10])

        y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
    return y_conv, keep_prob

def conv2d(x, W):
    """conv2d returns a 2d convolution layer with full stride."""
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding="SAME")

def max_pool_2x2(x):
    """max_pool_2x2 downsamples a feature map by 2X."""
    return tf.nn.max_pool(
        x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding="SAME")

def weight_variable(shape):
    """weight_variable generates a weight variable of a given shape."""
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    """bias_variable generates a bias variable of a given shape."""
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

class TrainMNIST(Trainable):
    """Example MNIST trainable."""

    def _setup(self, config):
        global activation_fn

        self.timestep = 0

        # Import data
        for _ in range(10):
            try:
                self.mnist = input_data.read_data_sets(
                    "/tmp/mnist_ray_demo", one_hot=True)
                break
            except Exception as e:
                print("Error loading data, retrying", e)
                time.sleep(5)

        assert self.mnist

        self.x = tf.placeholder(tf.float32, [None, 784])
        self.y_ = tf.placeholder(tf.float32, [None, 10])

        activation_fn = getattr(tf.nn, config["activation"])

        # Build the graph for the deep net
        y_conv, self.keep_prob = setupCNN(self.x)

        with tf.name_scope("loss"):
            cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
                labels=self.y_, logits=y_conv)
        cross_entropy = tf.reduce_mean(cross_entropy)

        with tf.name_scope("adam_optimizer"):
            train_step = tf.train.AdamOptimizer(
                config["learning_rate"]).minimize(cross_entropy)

        self.train_step = train_step

        with tf.name_scope("accuracy"):
            correct_prediction = tf.equal(
                tf.argmax(y_conv, 1), tf.argmax(self.y_, 1))
            correct_prediction = tf.cast(correct_prediction, tf.float32)
        self.accuracy = tf.reduce_mean(correct_prediction)

        self.sess = tf.Session()
        self.sess.run(tf.global_variables_initializer())
        self.iterations = 0
        self.saver = tf.train.Saver()

    def _train(self):
        for i in range(10):
            batch = self.mnist.train.next_batch(50)
            self.sess.run(
                self.train_step,
                feed_dict={
                    self.x: batch[0],
                    self.y_: batch[1],
                    self.keep_prob: 0.5
                })

        batch = self.mnist.train.next_batch(50)
        train_accuracy = self.sess.run(
            self.accuracy,
            feed_dict={
                self.x: batch[0],
                self.y_: batch[1],
                self.keep_prob: 1.0
            })

        self.iterations += 1
        return {"mean_accuracy": train_accuracy}

    def _save(self, checkpoint_dir):
        prefix = self.saver.save(
            self.sess, checkpoint_dir + "/save", global_step=self.iterations)
        return {"prefix": prefix}

    def _restore(self, ckpt_data):
        prefix = ckpt_data["prefix"]
        return self.saver.restore(self.sess, prefix)

# !!! Example of using the ray.tune Python API !!!
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--smoke-test", action="store_true", help="Finish quickly for testing")
    args, _ = parser.parse_known_args()
    mnist_spec = {
        "stop": {
            "mean_accuracy": 0.99,
            "time_total_s": 600,
        },
        "config": {
            "learning_rate": sample_from(
                lambda spec: 10**np.random.uniform(-5, -3)),
            "activation": grid_search(["relu", "elu", "tanh"]),
        },
        "num_samples": 10,
    }

    if args.smoke_test:
        mnist_spec["stop"]["training_iteration"] = 20
        mnist_spec["num_samples"] = 2

    ray.init()
    # ray.init(redis_address="10.10.10.133:6379")
    hyperband = PopulationBasedTraining(
        time_attr="training_iteration",
        metric="mean_accuracy",
        mode="max",
        perturbation_interval=10,
        hyperparam_mutations={
            "learning_rate": lambda _: 10**np.random.uniform(-5, -3),
            "activation":["relu", "elu", "tanh"]
        }
        )

    tune.run(
        TrainMNIST,
        name="mnist_asy_pbt",
        scheduler=hyperband,
        **mnist_spec)
richardliaw commented 5 years ago

Can you try tune.run(reuse_actors=False..)?

zhaokang1228 commented 5 years ago

Can you try tune.run(reuse_actors=False..)?

I tried I tried tune.run(reuse_actors=False..), but there are still mistakes., but there are still mistakes.

2019-07-27 10:27:41,019 ERROR worker.py:1612 -- Possible unhandled error from worker: ray_TrainMNIST:restore_from_object() (pid=4542, host=kangkang-Vostro-3900)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/trainable.py", line 348, in restore_from_object
    self.restore(checkpoint_path)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/trainable.py", line 325, in restore
    self._restore(checkpoint_dict)
  File "/home/kangkang/PycharmProjects/experiment/mnist/tune_mnist_ray_pbt.py", line 208, in _restore
    return self.saver.restore(self.sess, prefix)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    if not checkpoint_management.checkpoint_exists(compat.as_text(save_path)):
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 372, in checkpoint_exists
    if file_io.get_matching_files(pathname):
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 363, in get_matching_files
    return get_matching_files_v2(filename)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 384, in get_matching_files_v2
    compat.as_bytes(pattern))
tensorflow.python.framework.errors_impl.NotFoundError: /home/kangkang/ray_results/mnist_asy_pbt/TrainMNIST_29_activation=tanh,learning_rate=4.3437e-05_2019-07-27_10-14-15qup2sz3o/tmpvk7s36vjsave_to_object/checkpoint_60; No such file or directory
richardliaw commented 5 years ago

Can you try the newest version of Ray (the latest snapshot of Ray master) and the examples? It looks like you're using an older version of the examples.

zhaokang1228 commented 5 years ago

Can you try the newest version of Ray (the latest snapshot of Ray master) and the examples? It looks like you're using an older version of the examples.

I downloaded a newer version. There is still an error when running ray0.7.2/python/ray/tune/examples/tune_mnist_ray_hyperband.py. erroe logs:

/home/kangkang/anaconda3/envs/kk/bin/python /home/kangkang/PycharmProjects/ray0.7.2/python/ray/tune/examples/tune_mnist_ray_hyperband.py
2019-07-27 21:06:37,755 INFO node.py:498 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-27_21-06-37_755615_27298/logs.
2019-07-27 21:06:37,862 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:47089 to respond...
2019-07-27 21:06:37,987 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:33280 to respond...
2019-07-27 21:06:37,988 INFO services.py:806 -- Starting Redis shard with 3.35 GB max memory.
2019-07-27 21:06:38,012 INFO node.py:512 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-27_21-06-37_755615_27298/logs.
2019-07-27 21:06:38,012 INFO services.py:1446 -- Starting the Plasma object store with 5.03 GB memory using /dev/shm.
2019-07-27 21:06:38,171 INFO tune.py:61 -- Tip: to resume incomplete experiments, pass resume='prompt' or resume=True to run()
2019-07-27 21:06:38,171 INFO tune.py:233 -- Starting a new experiment.
== Status ==
Using HyperBand: num_stopped=0 total_brackets=0
Round #0:
Resources requested: 0/8 CPUs, 0/0 GPUs
Memory usage on this node: 4.7/16.8 GB

WARNING: Logging before flag parsing goes to stderr.
W0727 21:06:38.371757 140107657627392 deprecation_wrapper.py:119] From /home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/logger.py:136: The name tf.VERSION is deprecated. Please use tf.version.VERSION instead.

W0727 21:06:38.372045 140107657627392 deprecation_wrapper.py:119] From /home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/logger.py:141: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

== Status ==
Using HyperBand: num_stopped=0 total_brackets=4
Round #0:
  Bracket(Max Size (n)=5, Milestone (r)=100, completed=0.0%): {PENDING: 4, RUNNING: 1} 
  Bracket(Max Size (n)=8, Milestone (r)=33, completed=0.0%): {PENDING: 8} 
  Bracket(Max Size (n)=15, Milestone (r)=11, completed=0.0%): {PENDING: 15} 
  Bracket(Max Size (n)=34, Milestone (r)=3, completed=0.0%): {PENDING: 12} 
Resources requested: 1/8 CPUs, 0/0 GPUs
Memory usage on this node: 4.8/16.8 GB
Result logdir: /home/kangkang/ray_results/mnist_hyperband_test
Number of trials: 40 ({'RUNNING': 1, 'PENDING': 39})
PENDING trials:
 - TrainMNIST_1_learning_rate=0.0007819:    PENDING
 - TrainMNIST_2_learning_rate=0.00016285:   PENDING
 - TrainMNIST_3_learning_rate=0.00018607:   PENDING
 - TrainMNIST_4_learning_rate=1.1748e-05:   PENDING
 - TrainMNIST_5_learning_rate=2.2345e-05:   PENDING
 - TrainMNIST_6_learning_rate=2.5542e-05:   PENDING
 - TrainMNIST_7_learning_rate=0.00034919:   PENDING
 - TrainMNIST_8_learning_rate=0.00037257:   PENDING
 - TrainMNIST_9_learning_rate=6.8584e-05:   PENDING
  ... 21 not shown
 - TrainMNIST_31_learning_rate=2.2145e-05:  PENDING
 - TrainMNIST_32_learning_rate=1.1461e-05:  PENDING
 - TrainMNIST_33_learning_rate=0.00073865:  PENDING
 - TrainMNIST_34_learning_rate=4.6341e-05:  PENDING
 - TrainMNIST_35_learning_rate=0.00038985:  PENDING
 - TrainMNIST_36_learning_rate=5.2181e-05:  PENDING
 - TrainMNIST_37_learning_rate=0.00054816:  PENDING
 - TrainMNIST_38_learning_rate=2.3224e-05:  PENDING
 - TrainMNIST_39_learning_rate=0.00011796:  PENDING
RUNNING trials:
 - TrainMNIST_0_learning_rate=0.00088512:   RUNNING
...
...
...
== Status ==
Using HyperBand: num_stopped=0 total_brackets=4
Round #0:
  Bracket(Max Size (n)=5, Milestone (r)=100, completed=23.2%): {RUNNING: 2, TERMINATED: 3} 
  Bracket(Max Size (n)=8, Milestone (r)=33, completed=17.8%): {PAUSED: 2, PENDING: 5, TERMINATED: 1} 
  Bracket(Max Size (n)=15, Milestone (r)=11, completed=4.5%): {PENDING: 11, RUNNING: 4} 
  Bracket(Max Size (n)=34, Milestone (r)=3, completed=0.0%): {PENDING: 10, RUNNING: 2} 
Resources requested: 8/8 CPUs, 0/0 GPUs
Memory usage on this node: 11.6/16.8 GB
Result logdir: /home/kangkang/ray_results/mnist_hyperband_test
Number of trials: 40 ({'TERMINATED': 4, 'RUNNING': 8, 'PAUSED': 2, 'PENDING': 26})
PAUSED trials:
 - TrainMNIST_5_learning_rate=2.2345e-05:   PAUSED, [1 CPUs, 0 GPUs], [pid=27348], 86 s, 33 iter, 0.78 acc
 - TrainMNIST_6_learning_rate=2.5542e-05:   PAUSED, [1 CPUs, 0 GPUs], [pid=27346], 84 s, 33 iter, 0.88 acc
PENDING trials:
 - TrainMNIST_8_learning_rate=0.00037257:   PENDING
 - TrainMNIST_9_learning_rate=6.8584e-05:   PENDING
 - TrainMNIST_10_learning_rate=0.00023818:  PENDING
  ... 20 not shown
 - TrainMNIST_37_learning_rate=0.00054816:  PENDING
 - TrainMNIST_38_learning_rate=2.3224e-05:  PENDING
 - TrainMNIST_39_learning_rate=0.00011796:  PENDING
RUNNING trials:
 - TrainMNIST_2_learning_rate=0.00016285:   RUNNING, [1 CPUs, 0 GPUs], [pid=27353], 87 s, 34 iter, 0.98 acc
 - TrainMNIST_4_learning_rate=1.1748e-05:   RUNNING, [1 CPUs, 0 GPUs], [pid=27347], 87 s, 34 iter, 0.7 acc
 - TrainMNIST_13_learning_rate=1.0998e-05:  RUNNING, [1 CPUs, 0 GPUs], [pid=27495], 18 s, 6 iter, 0.16 acc
  ... 2 not shown
 - TrainMNIST_16_learning_rate=0.00055211:  RUNNING, [1 CPUs, 0 GPUs], [pid=27479], 16 s, 5 iter, 0.9 acc
 - TrainMNIST_28_learning_rate=4.6834e-05:  RUNNING
 - TrainMNIST_29_learning_rate=0.00020737:  RUNNING
TERMINATED trials:
 - TrainMNIST_0_learning_rate=0.00088512:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27352], 50 s, 17 iter, 1 acc
 - TrainMNIST_1_learning_rate=0.0007819:    TERMINATED, [1 CPUs, 0 GPUs], [pid=27350], 52 s, 18 iter, 1 acc
 - TrainMNIST_3_learning_rate=0.00018607:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27349], 49 s, 16 iter, 1 acc
 - TrainMNIST_7_learning_rate=0.00034919:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27351], 53 s, 18 iter, 1 acc
...
...
...
2019-07-27 21:08:15,136 ERROR worker.py:1612 -- Possible unhandled error from worker: ray_TrainMNIST:save_to_object() (pid=27346, host=kangkang-Vostro-3900)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/trainable.py", line 283, in save_to_object
    checkpoint_prefix = self.save(tmpdir)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/trainable.py", line 252, in save
    checkpoint))
ValueError: The returned checkpoint path does not exist: /home/kangkang/ray_results/mnist_hyperband_test/TrainMNIST_6_learning_rate=2.5542e-05_2019-07-27_21-06-382qcwesyy/tmp0y94mj_vsave_to_object/checkpoint_33/save
...
...
...
== Status ==
Using HyperBand: num_stopped=0 total_brackets=4
Round #0:
  Bracket(Max Size (n)=5, Milestone (r)=100, completed=40.6%): {RUNNING: 1, TERMINATED: 4} 
  Bracket(Max Size (n)=8, Milestone (r)=33, completed=17.8%): {PAUSED: 2, PENDING: 5, TERMINATED: 1} 
  Bracket(Max Size (n)=15, Milestone (r)=11, completed=21.7%): {PAUSED: 10, PENDING: 5} 
  Bracket(Max Size (n)=12, Milestone (r)=12, completed=9.4%): {ERROR: 7, PAUSED: 1, PENDING: 4} 
Resources requested: 8/8 CPUs, 0/0 GPUs
Memory usage on this node: 5.6/16.8 GB
Result logdir: /home/kangkang/ray_results/mnist_hyperband_test
Number of trials: 40 ({'TERMINATED': 5, 'RUNNING': 1, 'PAUSED': 13, 'PENDING': 14, 'ERROR': 7})
ERROR trials:
 - TrainMNIST_28_learning_rate=4.6834e-05:  ERROR, 1 failures: /home/kangkang/ray_results/mnist_hyperband_test/TrainMNIST_28_learning_rate=4.6834e-05_2019-07-27_21-08-095dqreni3/error_2019-07-27_21-09-28.txt, [1 CPUs, 0 GPUs], [pid=27501], 5 s, 3 iter, 0.36 acc
 - TrainMNIST_29_learning_rate=0.00020737:  ERROR, 1 failures: /home/kangkang/ray_results/mnist_hyperband_test/TrainMNIST_29_learning_rate=0.00020737_2019-07-27_21-08-11lcfyh6cq/error_2019-07-27_21-09-38.txt, [1 CPUs, 0 GPUs], [pid=27482], 6 s, 3 iter, 0.72 acc
  ... 3 not shown
 - TrainMNIST_33_learning_rate=0.00073865:  ERROR, 1 failures: /home/kangkang/ray_results/mnist_hyperband_test/TrainMNIST_33_learning_rate=0.00073865_2019-07-27_21-08-25gflsa77z/error_2019-07-27_21-09-55.txt, [1 CPUs, 0 GPUs], [pid=28082], 7 s, 3 iter, 0.9 acc
 - TrainMNIST_34_learning_rate=4.6341e-05:  ERROR, 1 failures: /home/kangkang/ray_results/mnist_hyperband_test/TrainMNIST_34_learning_rate=4.6341e-05_2019-07-27_21-08-34v0nhtc7x/error_2019-07-27_21-09-58.txt, [1 CPUs, 0 GPUs], [pid=28156], 5 s, 3 iter, 0.32 acc
PAUSED trials:
 - TrainMNIST_5_learning_rate=2.2345e-05:   PAUSED, [1 CPUs, 0 GPUs], [pid=27348], 86 s, 33 iter, 0.78 acc
 - TrainMNIST_6_learning_rate=2.5542e-05:   PAUSED, [1 CPUs, 0 GPUs], [pid=27346], 84 s, 33 iter, 0.88 acc
  ... 9 not shown
 - TrainMNIST_22_learning_rate=0.00012578:  PAUSED, [1 CPUs, 0 GPUs], [pid=28556], 21 s, 11 iter, 0.9 acc
 - TrainMNIST_39_learning_rate=0.00011796:  PAUSED, [1 CPUs, 0 GPUs], [pid=28079], 24 s, 12 iter, 0.82 acc
PENDING trials:
 - TrainMNIST_8_learning_rate=0.00037257:   PENDING
 - TrainMNIST_9_learning_rate=6.8584e-05:   PENDING
  ... 10 not shown
 - TrainMNIST_37_learning_rate=0.00054816:  PENDING, [1 CPUs, 0 GPUs], [pid=28093], 5 s, 3 iter, 0.86 acc
 - TrainMNIST_38_learning_rate=2.3224e-05:  PENDING, [1 CPUs, 0 GPUs], [pid=28091], 5 s, 3 iter, 0.22 acc
RUNNING trials:
 - TrainMNIST_4_learning_rate=1.1748e-05:   RUNNING, [1 CPUs, 0 GPUs], [pid=27347], 184 s, 97 iter, 0.86 acc
TERMINATED trials:
 - TrainMNIST_0_learning_rate=0.00088512:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27352], 50 s, 17 iter, 1 acc
 - TrainMNIST_1_learning_rate=0.0007819:    TERMINATED, [1 CPUs, 0 GPUs], [pid=27350], 52 s, 18 iter, 1 acc
 - TrainMNIST_2_learning_rate=0.00016285:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27353], 129 s, 59 iter, 1 acc
 - TrainMNIST_3_learning_rate=0.00018607:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27349], 49 s, 16 iter, 1 acc
 - TrainMNIST_7_learning_rate=0.00034919:   TERMINATED, [1 CPUs, 0 GPUs], [pid=27351], 53 s, 18 iter, 1 acc
...
...
...
2019-07-27 21:10:02,322 ERROR ray_trial_executor.py:211 -- Error starting runner for Trial TrainMNIST_35_learning_rate=0.00038985
Traceback (most recent call last):
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 209, in start_trial
    self._start_trial(trial, checkpoint)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 138, in _start_trial
    or trial._checkpoint.value is not None)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 99, in _setup_runner
    "Trial runner reuse requires reset_trial() to be "
ray.tune.error.AbortTrialExecution: Trial runner reuse requires reset_trial() to be implemented and return True.
2019-07-27 21:10:04,324 INFO ray_trial_executor.py:187 -- Destroying actor for trial TrainMNIST_35_learning_rate=0.00038985. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2019-07-27 21:10:04,325 WARNING util.py:64 -- The `start_trial` operation took 2.003997564315796 seconds to complete, which may be a performance bottleneck.
Traceback (most recent call last):
  File "/home/kangkang/PycharmProjects/ray0.7.2/python/ray/tune/examples/tune_mnist_ray_hyperband.py", line 241, in <module>
    **mnist_spec)
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/tune.py", line 255, in run
    runner.step()
  File "/home/kangkang/anaconda3/envs/kk/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 284, in step
    "There are paused trials, but no more pending "
ray.tune.error.TuneError: There are paused trials, but no more pending trials with sufficient resources.

Process finished with exit code 1
zhaokang1228 commented 5 years ago

Can you try the newest version of Ray (the latest snapshot of Ray master) and the examples? It looks like you're using an older version of the examples.

My Python module:

(kk) kangkang@kangkang-Vostro-3900:~$ pip list
Package                       Version     Location                   
----------------------------- ----------- ---------------------------
absl-py                       0.7.1       
actionlib                     1.11.13     
angles                        1.9.11      
astor                         0.8.0       
atari-py                      0.2.6       
atomicwrites                  1.3.0       
attrs                         19.1.0      
baselines                     0.1.5       
bayesian-optimization         1.0.1       
bleach                        1.5.0       
bondpy                        1.8.3       
camera-calibration            1.12.23     
camera-calibration-parsers    1.11.13     
catkin                        0.7.18      
certifi                       2019.6.16   
cffi                          1.12.3      
chardet                       3.0.4       
Click                         7.0         
cloudpickle                   1.2.1       
colorama                      0.4.1       
cv-bridge                     1.12.8      
cycler                        0.10.0      
Cython                        0.29.12     
decorator                     4.4.0       
diagnostic-analysis           1.9.3       
diagnostic-common-diagnostics 1.9.3       
diagnostic-updater            1.9.3       
dill                          0.3.0       
dynamic-reconfigure           1.5.50      
filelock                      3.0.12      
flatbuffers                   1.11        
funcsigs                      1.0.2       
future                        0.17.1      
gast                          0.2.2       
gazebo-plugins                2.5.19      
gazebo-ros                    2.5.19      
gencpp                        0.6.0       
geneus                        2.2.6       
genlisp                       0.4.16      
genmsg                        0.5.11      
gennodejs                     2.0.1       
genpy                         0.6.7       
glfw                          1.8.2       
gluoncv                       0.4.0.post0 
google-pasta                  0.1.7       
graphviz                      0.8.4       
grpcio                        1.22.0      
gym                           0.13.1      
h5py                          2.9.0       
html5lib                      0.9999999   
hyperopt                      0.1.2       
idna                          2.8         
image-geometry                1.12.8      
imageio                       2.5.0       
importlib-metadata            0.18        
interactive-markers           1.11.4      
joblib                        0.13.2      
Keras-Applications            1.0.8       
Keras-Preprocessing           1.1.0       
kiwisolver                    1.1.0       
laser-geometry                1.6.4       
lockfile                      0.12.2      
Markdown                      3.1.1       
matplotlib                    3.1.0       
message-filters               1.12.14     
mkl-fft                       1.0.12      
mkl-random                    1.0.2       
more-itertools                7.1.0       
mpi4py                        3.0.2       
mujoco-py                     1.50.1.68   
mxnet                         1.5.0       
networkx                      2.3         
numpy                         1.16.4      
opencv-python                 4.1.0.25    
packaging                     19.0        
pandas                        0.24.2      
Pillow                        6.1.0       
pip                           19.1.1      
pluggy                        0.12.0      
pluginlib                     1.11.3      
progressbar2                  3.42.0      
protobuf                      3.9.0       
psutil                        5.6.3       
py                            1.8.0       
pybullet                      2.5.3       
pybulletgym                   0.1         /home/kangkang/pybullet-gym
pycparser                     2.19        
pyglet                        1.3.2       
pymongo                       3.8.0       
pyparsing                     2.4.0       
PyRep                         1.0         
pytest                        5.0.1       
pytest-timeout                1.3.3       
python-dateutil               2.8.0       
python-qt-binding             0.3.4       
python-utils                  2.3.0       
pytz                          2019.1      
PyYAML                        5.1.1       
pyzmq                         18.0.2      
qt-dotgraph                   0.3.11      
qt-gui                        0.3.11      
qt-gui-cpp                    0.3.11      
qt-gui-py-common              0.3.11      
ray                           0.7.2       
redis                         3.2.1       
requests                      2.22.0      
resource-retriever            1.12.4      
rosbag                        1.12.14     
rosboost-cfg                  1.14.6      
rosclean                      1.14.6      
roscreate                     1.14.6      
rosgraph                      1.12.14     
roslaunch                     1.12.14     
roslib                        1.14.6      
roslint                       0.11.0      
roslz4                        1.12.14     
rosmake                       1.14.6      
rosmaster                     1.12.14     
rosmsg                        1.12.14     
rosnode                       1.12.14     
rosparam                      1.12.14     
rospy                         1.12.14     
rosservice                    1.12.14     
rostest                       1.12.14     
rostopic                      1.12.14     
rosunit                       1.14.6      
roswtf                        1.12.14     
rqt-action                    0.4.9       
rqt-bag                       0.4.12      
rqt-bag-plugins               0.4.12      
rqt-console                   0.4.8       
rqt-dep                       0.4.9       
rqt-graph                     0.4.9       
rqt-gui                       0.5.0       
rqt-gui-py                    0.5.0       
rqt-image-view                0.4.13      
rqt-launch                    0.4.8       
rqt-logger-level              0.4.8       
rqt-moveit                    0.5.7       
rqt-msg                       0.4.8       
rqt-nav-view                  0.5.7       
rqt-plot                      0.4.8       
rqt-pose-view                 0.5.8       
rqt-publisher                 0.4.8       
rqt-py-common                 0.5.0       
rqt-py-console                0.4.8       
rqt-reconfigure               0.4.10      
rqt-robot-dashboard           0.5.7       
rqt-robot-monitor             0.5.8       
rqt-robot-steering            0.5.9       
rqt-runtime-monitor           0.5.7       
rqt-rviz                      0.5.10      
rqt-service-caller            0.4.8       
rqt-shell                     0.4.9       
rqt-srv                       0.4.8       
rqt-tf-tree                   0.6.0       
rqt-top                       0.4.8       
rqt-topic                     0.4.10      
rqt-web                       0.4.8       
rviz                          1.12.17     
scikit-learn                  0.21.2      
scipy                         1.3.0       
sensor-msgs                   1.12.7      
setproctitle                  1.1.10      
setuptools                    41.0.1      
six                           1.12.0      
smach                         2.0.1       
smach-ros                     2.0.1       
smclib                        1.8.3       
tblib                         1.4.0       
tensorboard                   1.14.0      
tensorflow                    1.14.0      
tensorflow-estimator          1.14.0      
termcolor                     1.1.0       
tf                            1.11.9      
tf-conversions                1.11.9      
tf2-geometry-msgs             0.5.20      
tf2-kdl                       0.5.20      
tf2-py                        0.5.20      
tf2-ros                       0.5.20      
topic-tools                   1.12.14     
tornado                       6.0.3       
tqdm                          4.32.2      
urllib3                       1.25.3      
wcwidth                       0.1.7       
Werkzeug                      0.15.4      
wheel                         0.33.4      
wrapt                         1.11.2      
xacro                         1.11.3      
zipp                          0.5.2       
zmq                           0.0.0
richardliaw commented 5 years ago

The latest snapshot of master is on the installation page of the docs. You should be on 0.8.0.dev2.

On Sat, Jul 27, 2019 at 6:23 AM zhaokang1228 notifications@github.com wrote:

Can you try the newest version of Ray (the latest snapshot of Ray master) and the examples? It looks like you're using an older version of the examples.

My Python module:

(kk) kangkang@kangkang-Vostro-3900:~$ pip list Package Version Location

absl-py 0.7.1 actionlib 1.11.13 angles 1.9.11 astor 0.8.0 atari-py 0.2.6 atomicwrites 1.3.0 attrs 19.1.0 baselines 0.1.5 bayesian-optimization 1.0.1 bleach 1.5.0 bondpy 1.8.3 camera-calibration 1.12.23 camera-calibration-parsers 1.11.13 catkin 0.7.18 certifi 2019.6.16 cffi 1.12.3 chardet 3.0.4 Click 7.0 cloudpickle 1.2.1 colorama 0.4.1 cv-bridge 1.12.8 cycler 0.10.0 Cython 0.29.12 decorator 4.4.0 diagnostic-analysis 1.9.3 diagnostic-common-diagnostics 1.9.3 diagnostic-updater 1.9.3 dill 0.3.0 dynamic-reconfigure 1.5.50 filelock 3.0.12 flatbuffers 1.11 funcsigs 1.0.2 future 0.17.1 gast 0.2.2 gazebo-plugins 2.5.19 gazebo-ros 2.5.19 gencpp 0.6.0 geneus 2.2.6 genlisp 0.4.16 genmsg 0.5.11 gennodejs 2.0.1 genpy 0.6.7 glfw 1.8.2 gluoncv 0.4.0.post0 google-pasta 0.1.7 graphviz 0.8.4 grpcio 1.22.0 gym 0.13.1 h5py 2.9.0 html5lib 0.9999999 hyperopt 0.1.2 idna 2.8 image-geometry 1.12.8 imageio 2.5.0 importlib-metadata 0.18 interactive-markers 1.11.4 joblib 0.13.2 Keras-Applications 1.0.8 Keras-Preprocessing 1.1.0 kiwisolver 1.1.0 laser-geometry 1.6.4 lockfile 0.12.2 Markdown 3.1.1 matplotlib 3.1.0 message-filters 1.12.14 mkl-fft 1.0.12 mkl-random 1.0.2 more-itertools 7.1.0 mpi4py 3.0.2 mujoco-py 1.50.1.68 mxnet 1.5.0 networkx 2.3 numpy 1.16.4 opencv-python 4.1.0.25 packaging 19.0 pandas 0.24.2 Pillow 6.1.0 pip 19.1.1 pluggy 0.12.0 pluginlib 1.11.3 progressbar2 3.42.0 protobuf 3.9.0 psutil 5.6.3 py 1.8.0 pybullet 2.5.3 pybulletgym 0.1 /home/kangkang/pybullet-gym pycparser 2.19 pyglet 1.3.2 pymongo 3.8.0 pyparsing 2.4.0 PyRep 1.0 pytest 5.0.1 pytest-timeout 1.3.3 python-dateutil 2.8.0 python-qt-binding 0.3.4 python-utils 2.3.0 pytz 2019.1 PyYAML 5.1.1 pyzmq 18.0.2 qt-dotgraph 0.3.11 qt-gui 0.3.11 qt-gui-cpp 0.3.11 qt-gui-py-common 0.3.11 ray 0.7.2 redis 3.2.1 requests 2.22.0 resource-retriever 1.12.4 rosbag 1.12.14 rosboost-cfg 1.14.6 rosclean 1.14.6 roscreate 1.14.6 rosgraph 1.12.14 roslaunch 1.12.14 roslib 1.14.6 roslint 0.11.0 roslz4 1.12.14 rosmake 1.14.6 rosmaster 1.12.14 rosmsg 1.12.14 rosnode 1.12.14 rosparam 1.12.14 rospy 1.12.14 rosservice 1.12.14 rostest 1.12.14 rostopic 1.12.14 rosunit 1.14.6 roswtf 1.12.14 rqt-action 0.4.9 rqt-bag 0.4.12 rqt-bag-plugins 0.4.12 rqt-console 0.4.8 rqt-dep 0.4.9 rqt-graph 0.4.9 rqt-gui 0.5.0 rqt-gui-py 0.5.0 rqt-image-view 0.4.13 rqt-launch 0.4.8 rqt-logger-level 0.4.8 rqt-moveit 0.5.7 rqt-msg 0.4.8 rqt-nav-view 0.5.7 rqt-plot 0.4.8 rqt-pose-view 0.5.8 rqt-publisher 0.4.8 rqt-py-common 0.5.0 rqt-py-console 0.4.8 rqt-reconfigure 0.4.10 rqt-robot-dashboard 0.5.7 rqt-robot-monitor 0.5.8 rqt-robot-steering 0.5.9 rqt-runtime-monitor 0.5.7 rqt-rviz 0.5.10 rqt-service-caller 0.4.8 rqt-shell 0.4.9 rqt-srv 0.4.8 rqt-tf-tree 0.6.0 rqt-top 0.4.8 rqt-topic 0.4.10 rqt-web 0.4.8 rviz 1.12.17 scikit-learn 0.21.2 scipy 1.3.0 sensor-msgs 1.12.7 setproctitle 1.1.10 setuptools 41.0.1 six 1.12.0 smach 2.0.1 smach-ros 2.0.1 smclib 1.8.3 tblib 1.4.0 tensorboard 1.14.0 tensorflow 1.14.0 tensorflow-estimator 1.14.0 termcolor 1.1.0 tf 1.11.9 tf-conversions 1.11.9 tf2-geometry-msgs 0.5.20 tf2-kdl 0.5.20 tf2-py 0.5.20 tf2-ros 0.5.20 topic-tools 1.12.14 tornado 6.0.3 tqdm 4.32.2 urllib3 1.25.3 wcwidth 0.1.7 Werkzeug 0.15.4 wheel 0.33.4 wrapt 1.11.2 xacro 1.11.3 zipp 0.5.2 zmq 0.0.0

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/5284?email_source=notifications&email_token=ABCRZZK3U6E5BPBARKWYKFDQBRD3RA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26LL7I#issuecomment-515683837, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZJQTYMMT2ROP2X3PCLQBRD3RANCNFSM4IHDPYSA .

zhaokang1228 commented 5 years ago

The latest version of ray is 0.7.2 on github, I can't find version 0.8.0.

richardliaw commented 5 years ago

Hi, try this - https://ray.readthedocs.io/en/latest/installation.html

On Sat, Jul 27, 2019 at 8:25 PM zhaokang1228 notifications@github.com wrote:

The latest version of ray is 0.7.2 on github, I can't find version 0.8.0.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/5284?email_source=notifications&email_token=ABCRZZLWZO7KENE45KAE2QLQBUGRFA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26WSGI#issuecomment-515729689, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZIDQA23WYPJISCBOHDQBUGRFANCNFSM4IHDPYSA .

zhaokang1228 commented 5 years ago

Hi, try this - https://ray.readthedocs.io/en/latest/installation.html … On Sat, Jul 27, 2019 at 8:25 PM zhaokang1228 @.***> wrote: The latest version of ray is 0.7.2 on github, I can't find version 0.8.0. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5284?email_source=notifications&email_token=ABCRZZLWZO7KENE45KAE2QLQBUGRFA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26WSGI#issuecomment-515729689>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZIDQA23WYPJISCBOHDQBUGRFANCNFSM4IHDPYSA .

I installed version 0.7.2 through pip install -U ray, but I got an error with git cloneand the installation failed.

richardliaw commented 5 years ago

Can you use the links for “latest snapshot of master”? They should be 0.8.0.

On Sun, Jul 28, 2019 at 1:22 AM zhaokang1228 notifications@github.com wrote:

Hi, try this - https://ray.readthedocs.io/en/latest/installation.html … <#m5089391854427698781> On Sat, Jul 27, 2019 at 8:25 PM zhaokang1228 @.***> wrote: The latest version of ray is 0.7.2 on github, I can't find version 0.8.0. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5284 https://github.com/ray-project/ray/issues/5284?email_source=notifications&email_token=ABCRZZLWZO7KENE45KAE2QLQBUGRFA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26WSGI#issuecomment-515729689>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZIDQA23WYPJISCBOHDQBUGRFANCNFSM4IHDPYSA .

I installed version 0.7.2 through pip install -U ray, but I got an error with git clone and the installation failed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/5284?email_source=notifications&email_token=ABCRZZI33FRFB5XAKJLPZGTQBVJMZA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26Z3JY#issuecomment-515743143, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZPCWSBFOGT3WL5QRWLQBVJMZANCNFSM4IHDPYSA .

zhaokang1228 commented 5 years ago

Can you use the links for “latest snapshot of master”? They should be 0.8.0. On Sun, Jul 28, 2019 at 1:22 AM zhaokang1228 notifications@github.com wrote: … Hi, try this - https://ray.readthedocs.io/en/latest/installation.html … <#m5089391854427698781> On Sat, Jul 27, 2019 at 8:25 PM zhaokang1228 @.***> wrote: The latest version of ray is 0.7.2 on github, I can't find version 0.8.0. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5284 <#5284>?email_source=notifications&email_token=ABCRZZLWZO7KENE45KAE2QLQBUGRFA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26WSGI#issuecomment-515729689>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZIDQA23WYPJISCBOHDQBUGRFANCNFSM4IHDPYSA . I installed version 0.7.2 through pip install -U ray, but I got an error with git clone and the installation failed. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5284?email_source=notifications&email_token=ABCRZZI33FRFB5XAKJLPZGTQBVJMZA5CNFSM4IHDPYSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26Z3JY#issuecomment-515743143>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCRZZPCWSBFOGT3WL5QRWLQBVJMZANCNFSM4IHDPYSA .

There is an error in running pip install -e . --verbose。 error logs：

(py3) kangkang@kangkang-Vostro-3900:~/ray/python$ pip install -e . --verbose
Created temporary directory: /tmp/pip-ephem-wheel-cache-9efla44i
Created temporary directory: /tmp/pip-req-tracker-10n_p7bw
Created requirements tracker '/tmp/pip-req-tracker-10n_p7bw'
Created temporary directory: /tmp/pip-install-otc15tvl
Obtaining file:///home/kangkang/ray/python
...
...
...

Installing collected packages: six, numpy, pyarrow
    Successfully installed numpy-1.17.0 pyarrow-0.14.0.RAY six-1.12.0
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/numpy-1.17.0.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/numpy already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/pyarrow-0.14.0.RAY.dist-info already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/pyarrow already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/six.py already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/__pycache__ already exists. Specify --upgrade to force replacement.
    WARNING: Target directory /home/kangkang/ray/python/ray/pyarrow_files/bin already exists. Specify --upgrade to force replacement.
    + export PYTHON_BIN_PATH=/home/kangkang/anaconda3/envs/py3/bin/python
    + PYTHON_BIN_PATH=/home/kangkang/anaconda3/envs/py3/bin/python
    + '[' NO == YES ']'
    + '[' YES == YES ']'
    + /home/kangkang/bin/bazel build //:ray_pkg --verbose_failures
    Loading:
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    INFO: Call stack for the definition of repository 'com_github_grpc_grpc' which is a http_archive (rule definition at /home/kangkang/.cache/bazel/_bazel_kangkang/32b1ed6be4da93a5e90f1c3e1ba65d86/external/bazel_tools/tools/build_defs/repo/http.bzl:229:16):
     - /home/kangkang/ray/bazel/ray_deps_setup.bzl:105:5
     - /home/kangkang/ray/WORKSPACE:5:1
    ERROR: An error occurred during the fetch of repository 'com_github_grpc_grpc':
       java.io.IOException: Error downloading [https://github.com/grpc/grpc/archive/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz] to /home/kangkang/.cache/bazel/_bazel_kangkang/32b1ed6be4da93a5e90f1c3e1ba65d86/external/com_github_grpc_grpc/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz: connect timed out
    DEBUG: Rule 'com_github_nelhage_rules_boost' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556014830 +0800"
    DEBUG: Rule 'com_github_jupp0r_prometheus_cpp' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d0c773da8af3db99c543dd0413f4427d835170eddfd517bfeba104236a8d2c07"
    DEBUG: Rule 'com_github_checkstyle_java' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1552542575 +0800"
    ERROR: error loading package '': in /home/kangkang/ray/bazel/ray_deps_build_all.bzl: Encountered error while reading extension file 'bazel/grpc_deps.bzl': no such package '@com_github_grpc_grpc//bazel': java.io.IOException: Error downloading [https://github.com/grpc/grpc/archive/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz] to /home/kangkang/.cache/bazel/_bazel_kangkang/32b1ed6be4da93a5e90f1c3e1ba65d86/external/com_github_grpc_grpc/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz: connect timed out
    ERROR: error loading package '': in /home/kangkang/ray/bazel/ray_deps_build_all.bzl: Encountered error while reading extension file 'bazel/grpc_deps.bzl': no such package '@com_github_grpc_grpc//bazel': java.io.IOException: Error downloading [https://github.com/grpc/grpc/archive/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz] to /home/kangkang/.cache/bazel/_bazel_kangkang/32b1ed6be4da93a5e90f1c3e1ba65d86/external/com_github_grpc_grpc/76a381869413834692b8ed305fbe923c0f9c4472.tar.gz: connect timed out
    INFO: Elapsed time: 1.117s
    INFO: 0 processes.
    FAILED: Build did NOT complete successfully (0 packages loaded)
    FAILED: Build did NOT complete successfully (0 packages loaded)
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/kangkang/ray/python/setup.py", line 180, in <module>
        license="Apache 2.0")
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/kangkang/ray/python/setup.py", line 73, in run
        subprocess.check_call(command)
      File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '-p', '/home/kangkang/anaconda3/envs/py3/bin/python']' returned non-zero exit status 1.
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-10n_p7bw'
ERROR: Command "/home/kangkang/anaconda3/envs/py3/bin/python -c 'import setuptools, tokenize;__file__='"'"'/home/kangkang/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps" failed with error code 1 in /home/kangkang/ray/python/
Exception information:
Traceback (most recent call last):
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 414, in run
    use_user_site=options.use_user_site,
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    **kwargs
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 910, in install
    install_options, global_options, prefix=prefix,
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 766, in install_editable
    cwd=self.setup_py_dir,
  File "/home/kangkang/anaconda3/envs/py3/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "/home/kangkang/anaconda3/envs/py3/bin/python -c 'import setuptools, tokenize;__file__='"'"'/home/kangkang/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps" failed with error code 1 in /home/kangkang/ray/python/
richardliaw commented 5 years ago

https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master

No need to pip install -e .; just pip install the wheel

zhaokang1228 commented 5 years ago

https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master

No need to pip install -e .; just pip install the wheel

Thank you! It's running.