I created a new parameter type for my application:
class ScaledIntegerParameter(ScaledNumericParameter, IntegerParameter):
"""
An integer parameter that is searched on a
linear scale after normalization, but stored without scaling
"""
def __init__(self, name, min_value, max_value, scaling, **kwargs):
assert scaling > 0
kwargs['value_type'] = int
super(ScaledNumericParameter, self).__init__(
name, min_value, max_value, **kwargs)
self.scaling = scaling
def _scale(self, v):
return int(v / self.scaling)
def _unscale(self, v):
return int(v * self.scaling)
def legal_range(self, config):
return map(self._scale, NumericParameter.legal_range(self, config))
def search_space_size(self):
return self._scale(
super(ScaledIntegerParameter, self).search_space_size())
And initialized it using min_value = 1000000000, max_value = 6000000000, and scale = 1000000000. This, should effectively be treated as an IntegerParameter in the [1, 6] range. Unfortunately, Opentuner seems to somewhere generate and use a value that is less than min_value. I'm listing program output since I cannot figure out why none of the generated parameter values match the first configuration datum supplied to the client program. Even if it's somehow related to the small search space (I use the --no-dups flag), the configuration shouldn't be out of bounds.
[ 1s] INFO sparktuner.spark_tuner: Added config: <class 'sparktuner.spark_tuner.ScaledIntegerParameter'>: legal range [1, 6], search space size 5
[ 1s] INFO sparktuner.spark_tuner: Setting value: 1, unscaled: 1073741824
[ 1s] INFO sparktuner.spark_tuner: get_value: 1073741824
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 1, unscaled: 1073741824
[ 1s] INFO sparktuner.spark_tuner: get_value: 1073741824
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 1, unscaled: 1073741824
[ 1s] INFO sparktuner.spark_tuner: get_value: 1073741824
[ 1s] INFO sparktuner.spark_tuner: Setting value: 1, unscaled: 1073741824
[ 1s] INFO sparktuner.spark_tuner: get_value: 1073741824
[ 1s] INFO sparktuner.spark_tuner: Setting value: 4, unscaled: 4294967296
[ 1s] INFO sparktuner.spark_tuner: get_value: 4294967296
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 6, unscaled: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: get_value: 6442450944
[ 1s] INFO sparktuner.spark_tuner: Setting value: 5, unscaled: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: get_value: 5368709120
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 2, unscaled: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: get_value: 2147483648
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Setting value: 3, unscaled: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: get_value: 3221225472
[ 1s] INFO sparktuner.spark_tuner: Config dict: {'driver_memory': 104857600}
I created a new parameter type for my application:
And initialized it using min_value = 1000000000, max_value = 6000000000, and scale = 1000000000. This, should effectively be treated as an IntegerParameter in the [1, 6] range. Unfortunately, Opentuner seems to somewhere generate and use a value that is less than min_value. I'm listing program output since I cannot figure out why none of the generated parameter values match the first configuration datum supplied to the client program. Even if it's somehow related to the small search space (I use the
--no-dupsflag), the configuration shouldn't be out of bounds.