Validation error in Example_13

[wereii]

Where

Example_13_Detecting_Market_Making, 9th cell (# Finally, we're ready to start...)

What

• Win10 (latest)
• native python3.6.5 32-bit (v3.6.5:f59c0932b4)
• modules: pip freeze
• ran from inside of virtualenv

  Changes

• in cell # 2 compute_date = datetime.datetime.now() to get fresh data
• in cell # 4 date_range = pd.date_range(compute_date - datetime.timedelta(days=30), compute_date - datetime.timedelta(days=5)) shortened the date range by 5 days to not load latest but blank data in next step (it hangs then)

  Notes

• My intuition (stacktrace) tells me this is happening because of some changes within latest bravado or jsonschema packages but I was unable to get thru the evekit module and trace what is it caused by.

Retrieving 2019-01-05 14:19:13.264590...
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
<ipython-input-9-34f53c7c8078> in <module>
      6 order_book = OrderBook.get_data_frame(dates=[compute_date], types=liquid_type_map[region_id], regions=[region_id], 
      7                                       config=dict(local_storage=".", tree=True, skip_missing=True, 
----> 8                                                   fill_gaps=True, verbose=True))

~\Desktop\eve-market-strategies\code\book\evekit\marketdata\order_book.py in get_data_frame(dates, types, regions, config)
    483             if verbose:
    484                 print("Retrieving %s" % (str(next_date)), end="...")
--> 485             results.extend(OrderBook.get_day(next_date, types, regions, config))
    486             if verbose:
    487                 print("done")

~\Desktop\eve-market-strategies\code\book\evekit\marketdata\order_book.py in get_day(date, types, regions, config)
    446                 # Last chance, try the market service.  This will be very slow for large
    447                 # amounts of data.
--> 448                 values = OrderBook.__read_service__(date, types, regions)
    449             results.extend(values)
    450         # If still no data, then check whether we should complain

~\Desktop\eve-market-strategies\code\book\evekit\marketdata\order_book.py in __read_service__(target_date, types, regions)
    394                     try:
    395                         result, response = client.MarketData.book(typeID=next_type, regionID=next_region,
--> 396                                                                   date=str(current_time) + " UTC").result()
    397                         if response.status_code == 200:
    398                             snapshots.append(MarketSnapshot.__from_service__(result))

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado\client.py in __call__(self, **op_kwargs)
    245 
    246         request_params = construct_request(
--> 247             self.operation, request_options, **op_kwargs)
    248 
    249         http_client = self.operation.swagger_spec.http_client

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado\client.py in construct_request(operation, request_options, **op_kwargs)
    283             request[request_option] = request_options[request_option]
    284 
--> 285     construct_params(operation, request, op_kwargs)
    286 
    287     return request

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado\client.py in construct_params(operation, request, op_kwargs)
    306                 "{0} does not have parameter {1}"
    307                 .format(operation.operation_id, param_name))
--> 308         marshal_param(param, param_value, request)
    309 
    310     # Check required params and non-required params with a 'default' value

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado_core\param.py in marshal_param(param, value, request)
    122 
    123     if swagger_spec.config['validate_requests']:
--> 124         validate_schema_object(swagger_spec, param_spec, value)
    125 
    126     param_type = param_spec.get('type')

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado_core\validate.py in validate_schema_object(swagger_spec, schema_object_spec, value)
     51 
     52     if obj_type in SWAGGER_PRIMITIVES:
---> 53         validate_primitive(swagger_spec, schema_object_spec, value)
     54 
     55     elif obj_type == 'array':

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado_core\validate.py in scrubbed(*args, **kwargs)
     31                 e.message = '*** ' + e.message[len(str(e.instance)):]
     32                 e.instance = '***'
---> 33             reraise(*sys.exc_info())
     34 
     35     return scrubbed

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado_core\validate.py in scrubbed(*args, **kwargs)
     23     def scrubbed(*args, **kwargs):
     24         try:
---> 25             return func(*args, **kwargs)
     26         except jsonschema.ValidationError as e:
     27             if (

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\bravado_core\validate.py in validate_primitive(swagger_spec, primitive_spec, value)
     77         primitive_spec,
     78         format_checker=swagger_spec.format_checker,
---> 79         resolver=swagger_spec.resolver).validate(value)
     80 
     81 

c:\users\tomas\appdata\local\programs\python\python36-32\lib\site-packages\jsonschema\validators.py in validate(self, *args, **kwargs)
    128         def validate(self, *args, **kwargs):
    129             for error in self.iter_errors(*args, **kwargs):
--> 130                 raise error
    131 
    132         def is_type(self, instance, type):

ValidationError: 2048 is not of type 'integer'

Failed validating 'type' in schema:
    {'description': 'Market type for which order book will be retrieved',
     'in': 'query',
     'name': 'typeID',
     'required': True,
     'type': 'integer'}

On instance:
    2048

[deadlybulb]

My guess is this will work if you set 'validate_requests' to false when you create the bravado client. That would be code like this:

SwaggerClient.from_url("https://evekit-market.orbital.enterprises//swagger",
                                         config={'use_models': False, 'validate_responses': False, 'validate_requests': False})

I've had trouble in the past with bravado not properly implementing validation. In this case, however, it looks like some part of my library is sloppy about type conversion. I'll attempt to reproduce.

Thanks for the report!

[deadlybulb]

Thanks for raising this ticket! As a result of debugging, I've found other problems due to changes in pandas and matplotlib. You'll eventually hit those as well so I'll need to release a new notebook (and update many of the other examples as well). I will continue with the current code until I can reproduce your issue (I haven't yet).

That said, another fix is to set compute_date (cell 2) to something earlier than the current day. Two days before the current day should always be safe. One day before the current day can be safe depending on time of day (if you work on this after noon UTC, then the previous day is normally fine). If you set compute_date in this way, then you'll retrieve data from the archive (on Google Cloud Storage) rather than the live server. This will avoid Swagger and will also be much faster.

What you're trying to do should absolutely work against the live server using the current day as long as you backdate market history as you've done. However, it is almost always slower because the live server has to compute the book snapshots on the fly.

[deadlybulb]

I've just pushed an update to Example 13 which should run correctly with the latest pandas and matplotlib. I haven't yet reproduced the issue you saw, although I now suspect it is caused by the latest version of swagger-spec-validator. I'll know shortly.

For now, I suggest you run the example as follows. This will allow you to use recent data without having to rely on swagger:

# In cell 2, set compute date to a recent date
compute_date = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=2)
compute_date = compute_date.replace(hour=0, minute=0, second=0, microsecond=0)

If you use the version of the example I just uploaded, then no other changes should be required. Note that the list of items in cell 14 will be different because you are using more recent data. Likewise, the types analyzed in the remainder of the example will be probably be different as the most actively traded items these days are probably quite different than two years ago.

Hope this helps!

[deadlybulb]

Still trying to reproduce this but with no luck so far. The only real difference between your environment and mine at this point is:

1. I'm using conda for everything except bravado (which must be installed via pip)
2. I'm on a mac whereas you are on windows.

I do have access to a windows machine so I'll try to reproduce there next. If that doesn't work, I will make some defensive changes which should hopefully prevent the issue in the future.

[wereii]

Sorry for late reply, was busy. Thank you for this fast reply and work!

I tried some debugging for myself, it only happens when I try fetching current date. dates=[datetime.datetime.now()] It threw same thing even with this dates=[datetime.datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)].

I will now try your new version and report back if anything bad happens. 👌

[deadlybulb]

Finally managed to reproduce this on windows and disabling request validation does indeed fix it. I'm fairly certain the requests are actually correct, but if I can't figure it out I'll check in the version which disables request validation.

[deadlybulb]

Fix committed to head.