Error when using Dataset.upload

jsanz commented 5 years ago

Using develop, I'm getting a weird error with Dataset.upload method. It's working fine with other Dataframes, maybe the JSON types are messing the upload?

This is the error trace. You have below on the details the testing script.

$ python3 test.py                                                                                                                                                                                                                 [15:29:35]
/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/sql.py:229: UserWarning: Batch SQL job created with job_id: 1ecab127-585b-46fa-ab6e-94252c48101f
  warnings.warn('Batch SQL job created with job_id: {job_id}'.format(job_id=data['job_id']))
Traceback (most recent call last):
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/auth.py", line 147, in send
    response = super(APIKeyAuthClient, self).send(relative_path, http_method, **requests_args)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/pyrestcli/auth.py", line 36, in send
    return self.session.request(http_method, url, **requests_args)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/requests/adapters.py", line 469, in send
    for i in request.body:
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/sql.py", line 335, in _compress_chunks
    for chunk in chunk_generator:
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/cartoframes/datasets/dataset.py", line 254, in _rows
    if pd.isnull(val) or val is None:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/sql.py", line 379, in copyfrom
    stream=True)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/auth.py", line 149, in send
    raise CartoException(e)
carto.exceptions.CartoException: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 29, in <module>
    context=cc, schema=cc.get_default_schema())
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/cartoframes/datasets/dataset.py", line 165, in upload
    self._copyfrom(with_lnglat)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/cartoframes/datasets/dataset.py", line 241, in _copyfrom
    self._rows(self._df, [c for c in self._df.columns if c != 'cartodb_id'], with_lnglat, geom_col)
  File "/home/jsanz/desktop/geocamp/env/lib/python3.6/site-packages/carto/sql.py", line 384, in copyfrom
    raise CartoException(e)
carto.exceptions.CartoException: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Test script

### Testing script ```python import pandas as pd # Use Pandas to read a JSON of Citibike stations and their statuses stations_data = pd.read_json('https://gbfs.citibikenyc.com/gbfs/en/station_information.json') stations = pd.DataFrame(stations_data.data[0]) status_data = pd.read_json('https://gbfs.citibikenyc.com/gbfs/en/station_status.json') status = pd.DataFrame(status_data.data[0]) # Join the station and statuses together by 'station_id' station_status = pd.merge(stations,status,how='left', on='station_id') # Initialize cartoframes import os, cartoframes from cartoframes import Credentials USERNAME = os.getenv('CARTO_USER') APIKEY = os.getenv('CARTO_API_KEY') creds = Credentials(username=USERNAME, key=APIKEY) cc = cartoframes.CartoContext(creds=creds) # upload the dataframe from cartoframes.datasets import Dataset d = Dataset.from_dataframe(station_status) d.upload(table_name='cb_stations_status', if_exists="replace", with_lnglat=('lon','lat'), context=cc, schema=cc.get_default_schema()) ```