search

Esri / arcgis-python-api

Documentation and samples for ArcGIS API for Python
https://developers.arcgis.com/python/
Apache License 2.0
1.9k stars 1.1k forks source link

KeyError: 'type' when publishing CSV to Enterprise Portal #947

Closed UnderJollyRoger closed 3 years ago

UnderJollyRoger commented 3 years ago

I have a CSV file that I cannot publish to ArcGIS Enterprise Portal from a ArcGIS Pro Notebook. I have a stand-alone script that requests JSON from a URL and writes it to a CSV. The content has no special characters that I'm aware of. When opening the CSV from my stand-alone script, I encoded it to utf-8. But, when subsequently attempting to publish to the Portal I get the KeyError. I'm suspecting it's a character encoding issue in the CSV, but I'm in no way certain.

To Reproduce I'm not sure if this is necessary but here is my script that writes to the CSV. It's completely reproducible assuming you have the relevant modules.

import json,requests
import os,sys
import csv

#change file path to new directory
path_to_file = os.path.join(r'C:\Users\jpilbeam\Downloads', 'c19_Vaccine_Current.csv')
#idph url goes here
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'

##get json data from idph
response = requests.get(idph_data,verify=True)
#read the json response and keep the VaccineAdministration part
data = response.json()['VaccineAdministration']

#write to file
with open(path_to_file, 'w', newline='', encoding='UTF-8') as csvfile:
    f = csv.writer(csvfile) 
    #write the headers of the csv file
    f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated', 
                'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 'CommInventory',
                'TotalInventory', 'InventoryDate'])
    for elem in data:
        #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...)
        f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], 
                    elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], 
                    elem['PersonsFullyVaccinated'], elem['PersonsFullyVaccinatedChange'], 
                    elem['Report_Date'], elem['Population'], elem['PctVaccinatedPopulation'], 
                    elem['LHDReportedInventory'], elem['CommunityReportedInventory'], 
                    elem['TotalReportedInventory'], elem['InventoryReportDate']])>

In a Pro Notebook I then run the following code on the resulting CSV. It's when I try publishing that I get the KeyError.

from IPython.display import display
from arcgis.gis import GIS
import os
gis = GIS('Home') #uses current Notebook as workspace

csv_file = r'C:\Users\jpilbeam\Downloads\c19_Vaccine_Current.csv' #path to CSV
csv_item = gis.content.add({}, csv_file) #add CSV to Enterprise Portal
display(csv_item) #display it here 

#location parameter needs to be set to "none" when publishing to Enterprise Portal
params={"type":"csv","locationType":"none"} 
csv_item.publish(publish_parameters=params) #publish to Enterprise Portal

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
In  [7]:
Line 3:     csv_item.publish(publish_parameters=params) #publish to Enterprise Portal

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in publish:
Line 10353: elif not buildInitialCache and ret[0]['type'].lower() == 'image service':

KeyError: 'type'
--------------------------------------------------------------------------->

If it makes any difference I was able to publish only by opening the CSV in Notepad --> setting Encoding to utf-8 --> giving the filename a *.csv extension --> then saving. Needless to say, I don't intend to go this route. image

Expected behavior I expected the CSV to publish once I set the encoding to utf-8 in the with open() as csvfile part. I've attached my posts from GeoNet and StackExchange relating to this issue. It shows how I've come to this point.

Platform

Additional context My related posts on the issue: https://gis.stackexchange.com/questions/390282/encode-csv-to-utf-8-before-publishing-to-enterprise-portal https://community.esri.com/t5/arcgis-api-for-python-questions/publish-csv-to-enterprise-portal-errors/td-p/1035018 https://community.esri.com/t5/arcgis-api-for-python-questions/exception-job-failed-when-publishing-csv-item/m-p/1036154

achapkowski commented 3 years ago

Try the workflow below.

gis_dest = GIS(profile='your_online_profile',verify_cert=False, trust_env=True)
# data
fp = r"c:\temp\vaccine.csv"
# content manager
cm = gis_dest.content

# 1, add the CSV file
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'vaccine',
    'tags' : 'tags',
    'typeKeywords' : "CSV"
    },
       data=fp)
# 2.  analyze the CSV file to autogenerate the publish parameters
analyze_csv = cm.analyze(item=item)
pp = analyze_csv['publishParameters']
# 3.  Modify the publish parameters 
#      for tables ensure the `locationType` is None
#
pp['name'] = f'data_{uuid.uuid4().hex[:4]}'
pp['locationType'] = 'none'

# 3.  Publish
pitem = item.publish(publish_parameters=pp, file_type='csv')
print(pitem.tables)
UnderJollyRoger commented 3 years ago

I ran into an error on step 2. Up until this I've had no problems. I've added the CSV file to the Portal in step 1.

from IPython.display import display
from arcgis.gis import GIS
import os

gis_dest = GIS('https://gis.willcountyillinois.com/portal/home/', 'jpilbeam@willcountyillinois.com', 'joliet302',
verify_cert=False, trust_env=True)
# data
fp = r'C:\Users\jpilbeam\Downloads\c19_Vaccine_Current_test.csv'
# content manager
cm = gis_dest.content

# 1, add the CSV file
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'vaccine_test',
    'tags' : 'tags',
    'typeKeywords' : "CSV"
    },
       data=fp)

# 2.  analyze the CSV file to autogenerate the publish parameters
analyze_csv = cm.analyze(item=item)
pp = analyze_csv['publishParameters']
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
In  [13]:
Line 2:     analyze_csv = cm.analyze(item=item)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in analyze:
Line 4381:  return gis._con.post(path=surl, postdata=params, files=files)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in post:
Line 720:   force_bytes=kwargs.pop('force_bytes', False))

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in _handle_response:
Line 504:   data = resp.json()

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py, in json:
Line 900:   return complexjson.loads(self.text, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py, in loads:
Line 348:   return _default_decoder.decode(s)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in decode:
Line 337:   obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in raw_decode:
Line 355:   raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
---------------------------------------------------------------------------

From the Portal Administrator Directory: image

UnderJollyRoger commented 3 years ago

Using a different CSV I ran the provided script again. This time I got a different error. It failed on the Analyze call again. The 000735 error states there may be a parameter missing, but I'm unsure which.

from IPython.display import display
from arcgis.gis import GIS
import os

gis_dest = GIS('Home',verify_cert=False, trust_env=True)
# data
fp = r'C:\Users\jpilbeam\Downloads\c19_Vaccine_Current_test.csv' #path to CSV
# content manager
cm = gis_dest.content

# 1, add the CSV file
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'vaccine_test2',
    'tags' : 'tags',
    'typeKeywords' : "CSV"
    },
       data=fp)

# 2.  analyze the CSV file to autogenerate the publish parameters
analyze_csv = cm.analyze(item=item)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
In  [20]:
Line 2:     analyze_csv = cm.analyze(item=item)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in analyze:
Line 4381:  return gis._con.post(path=surl, postdata=params, files=files)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in post:
Line 720:   force_bytes=kwargs.pop('force_bytes', False))

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in _handle_response:
Line 514:   self._handle_json_error(data['error'], errorcode)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in _handle_json_error:
Line 536:   raise Exception(errormessage)

Exception: Analyze Service error: Server tool execution failed : ERROR 000735: File type: Value is required Failed.
(Error Code: 0)
---------------------------------------------------------------------------

By the way, I was able to publish the CSV as a hosted table in Portal. So, it must something in the script?

ex1 ex2

UnderJollyRoger commented 3 years ago

I did this, and it published:


analyze_csv = cm.analyze(item=item, file_type='csv') # <--added file type

# 3.  Modify the publish parameters 
#      for tables ensure the `locationType` is None
#
#pp['name'] = f'data_{uuid.uuid4().hex[:4]}' # I commented this out because it said uuid was not defined
pp['locationType'] = 'none'

# 3.  Publish
pitem = item.publish(publish_parameters=pp, file_type='csv')
print(pitem.tables)
[<Table url:"https://gis.willcountyillinois.com/hosting/rest/services/Hosted/c19_Vaccine_Current_test/FeatureServer/0">]
UnderJollyRoger commented 3 years ago

But, if I try publishing the CSV with the requested JSON (i.e. if I run the whole script) I still get the JSONDecodeError. Someone on GeoNet suggested I can access my server's REST API and perform add contents and analyze by sending a requests.post() to the urls. But, i'm not sure how to do that and can't find any examples.

achapkowski commented 3 years ago

You need to import uuid module in your script.

I don't understand what the requested JSON is. Maybe post your csv and web traffic (scrubbed of credentials) and we can assist.

The item.analyze is doing the analyze.

UnderJollyRoger commented 3 years ago

This will generate the CSV I want to publish to Enterprise Portal as a hosted table:

import json,requests
import os,sys
import csv
from datetime import datetime
from arcgis.gis import GIS
import uuid

#file name
csv_file = 'C19VaccineCurrent'
#the directory you want the csv written to
file_path = r'C:\where\you\want\it\saved'
##idph url goes here
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'
headers = {'user-agent': 'Mozilla/5.0'}

full_file = os.path.join(file_path, csv_file) + str(datetime.now().strftime('_%m_%d_%Y_%H%M')) + '.csv'

#get json data from idph
response = requests.get(idph_data, headers=headers, verify=True)
#read the json response and keep the VaccineAdministration part
data = response.json()['VaccineAdministration']

#write to file
with open(full_file, 'w', newline='', encoding='UTF-8') as csvfile:
    f = csv.writer(csvfile) 
    #write the headers of the csv file
    f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated', 
                'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 
                'CommInventory','TotalInventory', 'InventoryDate'])
    for elem in data:
        #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...)
        f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], 
                    elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], elem['PersonsFullyVaccinated'], 
                    elem['PersonsFullyVaccinatedChange'], elem['Report_Date'], elem['Population'], 
                    elem['PctVaccinatedPopulation'], elem['LHDReportedInventory'], 
                    elem['CommunityReportedInventory'], elem['TotalReportedInventory'], 
                    elem['InventoryReportDate']])

I ran the script you gave me, and this time I imported the uuid module:


gis_dest = GIS('Home', verify_cert=False, trust_env=True) 
# data
fp = r"C:\path\to\Test\C19VaccineCurrent_03_31_2021_1039.csv"
# content manager
cm = gis_dest.content

# 1.  add the CSV file
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'VaccineCurrent',
    'tags' : 'tags',
    'typeKeywords' : "CSV"
    },
       data=full_file)
# 2.  analyze the CSV file to autogenerate the publish parameters
analyze_csv = cm.analyze(item=item)
pp = analyze_csv['publishParameters']
#      Modify the publish parameters 
#      for tables ensure the `locationType` is None
#
pp['name'] = f'data_{uuid.uuid4().hex[:4]}'
pp['locationType'] = 'none'

# 3.  Publish
pitem = item.publish(publish_parameters=pp, file_type='csv')
print(pitem.tables)

It gave me the following error:

JSONDecodeError                           Traceback (most recent call last)
In  [25]:
Line 14:    data=full_file)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in add:
Line 4265:  owner_name, folder)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_portalpy.py, in add_item:
Line 355:   resp = self.con.post(path, postdata, files)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in post:
Line 720:   force_bytes=kwargs.pop('force_bytes', False))

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in _handle_response:
Line 504:   data = resp.json()

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py, in json:
Line 900:   return complexjson.loads(self.text, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py, in loads:
Line 348:   return _default_decoder.decode(s)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in decode:
Line 337:   obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in raw_decode:
Line 355:   raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

If you go by the error and look at line 1 column 1 in the CSV it is a text field with the value 'County'. I've been trying to use Fiddler 4 to view the web traffic while running the script from a Pro Notebook, but when I have Fiddler open I get this error:

response = requests.get(idph_data,headers=headers, verify=True) #this is the line it errors on

ValueError: check_hostname requires server_hostname
achapkowski commented 3 years ago

You are not passing in the f=json parameter needed to get a JSON response from the REST API. If you want to work directly with the REST API, I suggest you take a look here: https://developers.arcgis.com/rest/

UnderJollyRoger commented 3 years ago

Thanks for your help. I'm not familiar enough with ArcGIS API for Python to get through this without millions of questions. I'm re-posting the current script I'm working with because I noticed I had an extra variable in the last one.

#This is a two-part script. The first part will write the requested JSON to a CSV. The second part will add 
#then publish the CSV to the Portal.

import json, requests
from arcgis.gis import GIS
import os,sys
import csv
from datetime import datetime
import uuid

#file name
csv_file = 'C19VaccineCurrent'
#the directory you want the csv written to
file_path = r'C:\Users\Test'
##idph url goes here
idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'
#headers
headers = {'user-agent': 'Mozilla/5.0'}

#The following section writes the requested JSON to CSV

full_file = os.path.join(file_path, csv_file) + str(datetime.now().strftime('_%m_%d_%Y_%H%M')) + '.csv'

#get json data from idph
response = requests.get(idph_data, headers=headers, verify=True)
#read the json response and keep the VaccineAdministration part
data = response.json()['VaccineAdministration']

#write to file
with open(full_file, 'w', newline='', encoding='UTF-8') as csvfile:
    f = csv.writer(csvfile) 
    #write the headers of the csv file
    f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated', 
                'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 
                'CommInventory','TotalInventory', 'InventoryDate'])
    for elem in data:
        #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...)
        f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], 
                    elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], elem['PersonsFullyVaccinated'], 
                    elem['PersonsFullyVaccinatedChange'], elem['Report_Date'], elem['Population'], 
                    elem['PctVaccinatedPopulation'], elem['LHDReportedInventory'], 
                    elem['CommunityReportedInventory'], elem['TotalReportedInventory'], 
                    elem['InventoryReportDate']])

#This section takes the CSV and publishes it to Enterprise Portal. Enter the relevant Enterprise Login info 
#and if neccessary give the item a title and tag(s).

gis_dest = GIS('Home', verify_cert=False, trust_env=True) 
# data

# content manager
cm = gis_dest.content

# 1. add the CSV file
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'C19VaccineCurrent',
    'tags' : 'tags',
    'typeKeywords' : "CSV"
    },
       data=full_file)

# 2.  analyze the CSV file to autogenerate the publish parameters
analyze_csv = cm.analyze(item=item)
pp = analyze_csv['publishParameters']
#    Modify the publish parameters 
#    for tables ensure the `locationType` is None
#
pp['name'] = f'data_{uuid.uuid4().hex[:4]}'
pp['locationType'] = 'none'

# 3.  Publish
pitem = item.publish(publish_parameters=pp, file_type='csv')
print(pitem.tables)

As it is now it will throw the JSONDecodeError on the data=full_file line. Thanks, I'm looking into where the f=json parameter is supposed to go.

Detteor commented 3 years ago

After seeing the issue on ESRI community forum, I wanted to show you how I am handling the issue. You can still publish without analyzing it, if you are happy with your data. Also, you can use pandas DataFrames to manage/change your data. Another solution using the 'ContentManager' class is to use the method 'import_data()', but this is limited to a 1,000 rows.

I am hoping someone can provide a reason why it won't analyze the file using the ContentManager Class.

If you want to see how the parameters pertain to REST API: Analyze Publish Item


import pandas as pd
from arcgis import GIS, features
from datetime import datetime as dt
import requests

now = dt.now()
dStr = now.strftime('_%m_%d_%Y')

#Connection to AGOL
gis = GIS(url='URL', username= 'username', password='password')

#Connection to ArcGIS Enterprise Portal running 10.5.1
gisE = GIS(url='URL', username='username', password='password')

j = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'

r = requests.get(j).json()['VaccineAdministration']

csvData = f'/media/WDB/Backup/Projects/data/VaccineInformation{dStr}.csv'

#Create DataFrame from the json
df = pd.json_normalize(r)
df.to_csv(csvData, index=False)

#create a spatially enabled dataframe. You will need to specify the Spatial Reference using 'sr=WKID' parameter.
uDF = features.GeoAccessor.from_xy(df, 'Latitude', 'Longitude')
#Create Feature Layer on ArcGIS Enterprise Portal.
uDF.spatial.to_featurelayer(f'Vaccine Information{dStr}', gisE)

cm = gis.content

#Add CSV to ArcGIS Enterprise if you want to.
item = cm.add(item_properties={
    'type' : 'CSV',
    'title' : 'VaccineCurrent',
    'tags' : 'test',
    'typeKeywords' : 'CSV'}, data=csvData)```
UnderJollyRoger commented 3 years ago

@Detteor Thanks for the input. I ran what you have from a Pro Notebook once and I was successful from start to finish (with some modifications). I ran the same thing again. Fails. Why so fragile?

import pandas as pd
from arcgis import GIS, features
from datetime import datetime as dt
import requests
from arcgis.features import GeoAccessor, GeoSeriesAccessor

now = dt.now()
dStr = now.strftime('_%m_%d_%Y')

#Connection to ArcGIS Enterprise Portal running 10.5.1
gisE = GIS('Home')

j = r'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent'

r = requests.get(j).json()['VaccineAdministration']

csvData = f'C:/Users/jpilbeam/Test{dStr}.csv'

#Create DataFrame from the json
df = pd.json_normalize(r)
df.to_csv(csvData, index=False)

#create a spatially enabled dataframe. You will need to specify the Spatial Reference using 'sr=WKID' parameter.
uDF = features.GeoAccessor.from_xy(df, 'Latitude', 'Longitude')
#Create Feature Layer on ArcGIS Enterprise Portal.
uDF.spatial.to_featurelayer(f'Vaccine Information{dStr}', gisE)
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
In  [134]:
Line 4:     uDF.spatial.to_featurelayer(f'Vaccine Information{dStr}', gisE)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\features\geo\_accessor.py, in to_featurelayer:
Line 2182:  return content.import_data(self._data, folder=folder, title=title, tags=tags)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in import_data:
Line 5287:  folder=folder)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py, in add:
Line 4265:  owner_name, folder)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_portalpy.py, in add_item:
Line 355:   resp = self.con.post(path, postdata, files)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in post:
Line 720:   force_bytes=kwargs.pop('force_bytes', False))

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py, in _handle_response:
Line 504:   data = resp.json()

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py, in json:
Line 900:   return complexjson.loads(self.text, **kwargs)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py, in loads:
Line 348:   return _default_decoder.decode(s)

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in decode:
Line 337:   obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py, in raw_decode:
Line 355:   raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value: line 1 column 1 (char 0)
---------------------------------------------------------------------------
Detteor commented 3 years ago

@UnderJollyRoger At this point, I would try to install a new environment for python. I have never had very good luck with using the default environment created by ArcGIS Pro. I personally use anaconda to create new environments and to install new packages.

Once you have a new environment setup, if you have ArcGIS Pro 2.7 installed, you can use conda install -c esri arcgispro (this will install a metapackage that makes it so that your packages won't become incompatible with other ESRI packages) and conda install -c esri arcpy (this installs arcpy, but requires it to be installed on the same machine). After that, you can conda install -c esri arcgis.

Some resources and references ESRI Conda Install Docs Anaconda Main Site

UnderJollyRoger commented 3 years ago

@Detteor Good to know. I'm already using a conda environment in Pro 2.7.2 as fas as I can tell. I didn't bother going through the Anaconda site, as I installed the API with Pro.

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>conda info --envs
# conda environments:
#
arcgispro-py3         *  C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3
root                     C:\Program Files\ArcGIS\Pro\bin\Python

When I did the install commands the Python API was updated, but arcpy 2.7 was already installed. 

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>conda install -c esri arcgispro
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3:

The following packages will be UPDATED:

    arcgis:    1.8.4-py37_1694          esri --> 1.8.5-py37_1775         esri
    decorator: 4.4.2-pyhd3eb1b0_0            --> 5.0.5-pyhd3eb1b0_0
    regex:     2021.3.17-py37h2bbff1b_0      --> 2021.4.4-py37h2bbff1b_0
    sqlite:    3.35.2-h2bbff1b_0             --> 3.35.4-h2bbff1b_0
    terminado: 0.9.3-py37haa95532_0          --> 0.9.4-py37haa95532_0

(arcgispro-py3) C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3>conda install -c esri arcpy
Fetching package metadata .............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3:
#
arcpy                     2.7             py37_arcgispro_26810  [arcgispro]  esri

I'm connected to the same environment in my IDE. This is from Visual Studio 2019. image

UnderJollyRoger commented 3 years ago

@Detteor By the way, this is from an ESRI blog: it is not recommended that you have both Anaconda and ArcGIS Pro installed on a single machine

Detteor commented 3 years ago

@UnderJollyRoger

@Detteor By the way, this is from an ESRI blog: it is not recommended that you have both Anaconda and ArcGIS Pro installed on a single machine

This Definitely might have been the case when using ArcGIS Pro 2.2, but now with the metapackages and being able to install arcpy in an environment it doesn't seem as impactful anymore. ENV and PATH variables are always a pain in windows, so I don't even let conda set its own variables. Also, they don't want you to mess with the base install since that can cause a lot of issues, this is one of the reasons that they don't let you change the base install for ArcGIS Pro.

@Detteor Good to know. I'm already using a conda environment in Pro 2.7.2 as fas as I can tell. I didn't bother going through the Anaconda site, as I installed the API with Pro. I'm connected to the same environment in my IDE. This is from Visual Studio 2019.

When I start to work on new projects, I always use a new environment that has the latest package versions. I actually never touch the conda environment that comes with ArcGIS Pro since I have found that it causes a lot of problems. I do now that the version of ArcGIS API that comes with ArcGIS Pro is locked at 1.8.3 due to compatibility issues. You might want to create a whole new environment that isn't going to be used in ArcGIS Pro and that will only be used in your IDE.

You can create an env.yml file that you can use to install the packages that you want in your new environment. You will need to activate your environment and type conda env export > env.yml, this will save a yml file in your current working directory of the packages currently installed. You can also use conda env export --from-history > env.yml, this will export only the packages that you have installed.Then you can use conda create -n myenv python=3.7 use 3.7 since that is what arcpy is running in ArcGIS Pro.

UnderJollyRoger commented 3 years ago

@Detteor Thanks a lot for the knowledge transfer! So, using Anaconda I created an environment which has the following packages. From VS Code I then ran the script you provided me. It's running from env1, and I assume it has all the relevant modules. It runs until it hits the JSONDecodeError on uDF.spatial.to_featurelayer(f'Vaccine Information{dStr}', gisE). That's where it was failing on the most recent script I posted above.

So, that script you provided me runs for you, and you're able to publish a CSV to Enterprise Portal? I wonder what I'm missing?

(base) C:\Users\jpilbeam>conda list -n env1
# packages in environment at C:\Users\jpilbeam\Miniconda3\envs\env1:
#
# Name                    Version                   Build  Channel
anyio                     2.2.0            py37haa95532_0
appdirs                   1.4.4                      py_0
arcgis                    1.8.5                 py37_1775    esri
arcgispro                 2.7                           0    esri
arcpy                     2.7             py37_arcgispro_26810  [arcgispro]  esri
argon2-cffi               20.1.0           py37h2bbff1b_1
asn1crypto                1.4.0                      py_0
atomicwrites              1.4.0                      py_0
attrs                     20.3.0             pyhd3eb1b0_0
babel                     2.9.0              pyhd3eb1b0_0
backcall                  0.2.0              pyhd3eb1b0_0
black                     19.10b0                    py_0
blas                      1.0                         mkl
bleach                    3.3.0              pyhd3eb1b0_0
blinker                   1.4              py37haa95532_0
brotlipy                  0.7.0           py37h2bbff1b_1003
ca-certificates           2021.1.19            haa95532_1
cached-property           1.5.2                      py_0
certifi                   2020.12.5        py37haa95532_0
cffi                      1.14.5           py37hcd4344a_0
cftime                    1.0.0b1                  py37_0    esri
chardet                   4.0.0           py37haa95532_1003
click                     7.1.2              pyhd3eb1b0_0
colorama                  0.4.4              pyhd3eb1b0_0
cppzmq                    4.4.1                         2    esri
cryptography              2.8                      py37_0    esri
cycler                    0.10.0                   py37_0
decorator                 5.0.5              pyhd3eb1b0_0
defusedxml                0.7.1              pyhd3eb1b0_0
despatch                  0.1.0                    py37_0    esri
entrypoints               0.3                      py37_0
et_xmlfile                1.0.1                   py_1001
fastcache                 1.1.0            py37he774522_0
flake8                    3.9.0              pyhd3eb1b0_0
freetype                  2.10.1                   vc14_0  [vc14]  esri
future                    0.18.2                   py37_0    esri
gdal                      2.3.3           arcgispro_py37_16713  [arcgispro]  esri
h5py                      2.10.0          py37_arcgispro_9  [arcgispro]  esri
html5lib                  1.1                        py_0
icc_rt                    2019.0.5            arcgispro_0  [arcgispro]  esri
idna                      2.10               pyhd3eb1b0_0
importlib-metadata        3.7.3            py37haa95532_1
importlib_metadata        3.7.3                hd3eb1b0_1
iniconfig                 1.1.1              pyhd3eb1b0_0
intel-openmp              2020.0            arcgispro_166  [arcgispro]  esri
ipykernel                 5.3.4            py37h5ca1d4c_0
ipython                   7.18.1                   py37_0    esri
ipython_genutils          0.2.0              pyhd3eb1b0_1
ipywidgets                7.6.3              pyhd3eb1b0_1
jdcal                     1.4.1                      py_0
jedi                      0.18.0           py37haa95532_1
jinja2                    2.11.3             pyhd3eb1b0_0
jpeg                      9d                            0    esri
json5                     0.9.4                    py37_0    esri
jsonschema                3.2.0                      py_2
jupyter-packaging         0.7.12             pyhd3eb1b0_0
jupyter_client            6.1.7                      py_0    esri
jupyter_console           6.2.0                      py_2    esri
jupyter_contrib_core      0.3.3                    py37_3    esri
jupyter_contrib_nbextensions 0.5.1                   py37_10    esri
jupyter_core              4.6.3                    py37_2    esri
jupyter_highlight_selected_word 0.2.0                    py37_2    esri
jupyter_latex_envs        1.4.4                    py37_1    esri
jupyter_nbextensions_configurator 0.4.1                    py37_1    esri
jupyter_server            1.4.1            py37haa95532_0
jupyterlab                3.0.11             pyhd3eb1b0_1
jupyterlab_server         2.4.0              pyhd3eb1b0_0
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1
keyring                   21.4.0                   py37_0    esri
kiwisolver                1.3.1            py37hd77b12b_0
lerc                      2.2                        py_0    esri
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.18                        1    esri
libtiff                   4.1.0                         0    esri
libxml2                   2.9.10              arcgispro_0  [arcgispro]  esri
libxslt                   1.1.34               he774522_0
lxml                      4.6.3            py37h9b66d53_0
lz4-c                     1.9.3                h2bbff1b_0
markupsafe                1.1.1            py37hfa6e2cd_1
matplotlib                3.3.1           py37_arcgispro_0  [arcgispro]  esri
mccabe                    0.6.1                    py37_1
mistune                   0.8.4           py37hfa6e2cd_1001
mkl                       2020.0            arcgispro_167  [arcgispro]  esri
mkl-service               2.3.0            py37h196d8e1_0
mkl_fft                   1.3.0            py37h46781fe_0
mkl_random                1.2.0                    py37_0    esri
mpmath                    1.2.1            py37haa95532_0
mypy_extensions           0.4.3                    py37_0
nbclassic                 0.2.6              pyhd3eb1b0_0
nbconvert                 5.6.1                    py37_0    esri
nbformat                  5.0.7                      py_1    esri
netcdf4                   1.5.4           py37_arcgispro_5  [arcgispro]  esri
networkx                  2.5                      py37_0    esri
nlohmann_json             3.7.0                         1    esri
nose                      1.3.7           pyhd3eb1b0_1006
notebook                  6.1.4                    py37_1    esri
ntlm-auth                 1.4.0                      py_0    esri
numexpr                   2.7.3            py37hcbcaa1e_0
numpy                     1.19.2           py37hadc3359_0
numpy-base                1.19.2           py37ha3acd2a_0
oauthlib                  3.1.0                      py_0
olefile                   0.46                     py37_0
openpyxl                  3.0.7              pyhd3eb1b0_0
openssl                   1.1.1k               h2bbff1b_0
packaging                 20.9               pyhd3eb1b0_0
pandas                    1.2.3            py37hf11a4ad_0
pandocfilters             1.4.3            py37haa95532_1
parso                     0.8.2              pyhd3eb1b0_0
pathspec                  0.7.0                      py_0
pefile                    2019.4.18                  py_0
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow-simd               7.1.2                    py37_3    esri
pip                       21.0.1           py37haa95532_0
pluggy                    0.13.1           py37haa95532_0
pro_notebook_integration  2.7                      py37_1    esri
prometheus_client         0.8.0                      py_0    esri
prompt_toolkit            3.0.5                      py_0    esri
psutil                    5.8.0            py37h2bbff1b_1
py                        1.10.0             pyhd3eb1b0_0
pybind11                  2.3.0                         1    esri
pycodestyle               2.6.0              pyhd3eb1b0_0
pycparser                 2.20                       py_2
pyflakes                  2.2.0              pyhd3eb1b0_0
pygments                  2.7.0                      py_0    esri
pyjwt                     2.0.1            py37haa95532_0
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7              pyhd3eb1b0_0
pypdf2                    1.26.0                     py_2    esri
pyrsistent                0.17.3           py37he774522_0
pyshp                     2.1.3              pyhd3eb1b0_0
pysocks                   1.7.1                    py37_1
pytest                    6.1.1                    py37_0    esri
python                    3.7.10               h6244533_0
python-certifi-win32      1.2                      py37_0    esri
python-dateutil           2.8.1              pyhd3eb1b0_0
pytz                      2020.1                   py37_0    esri
pywin32                   227              py37he774522_1
pywin32-ctypes            0.2.0                    pypi_0    pypi
pywin32-security          228                      py37_3    esri
pywinpty                  0.5.7                    py37_0    esri
pyyaml                    5.4.1            py37h2bbff1b_1
pyzmq                     19.0.2                   py37_1    esri
regex                     2021.4.4         py37h2bbff1b_0
requests                  2.25.1             pyhd3eb1b0_0
requests-kerberos         0.12.0                        0    esri
requests-negotiate-sspi   0.5.2                    py37_1    esri
requests-oauthlib         1.3.0                      py_0
requests-toolbelt         0.9.1                      py_0
requests_ntlm             1.1.0                      py_0    esri
scipy                     1.6.2            py37h14eb087_0
send2trash                1.5.0              pyhd3eb1b0_1
setuptools                52.0.0           py37haa95532_0
simplegeneric             0.8.1                    py37_2
six                       1.15.0           py37haa95532_0
sniffio                   1.2.0            py37haa95532_1
sqlite                    3.35.4               h2bbff1b_0
sympy                     1.5.1                    py37_0    esri
terminado                 0.9.4            py37haa95532_0
testpath                  0.4.4              pyhd3eb1b0_0
toml                      0.10.2             pyhd3eb1b0_0
tornado                   6.1              py37h2bbff1b_0
traitlets                 5.0.5              pyhd3eb1b0_0
typed-ast                 1.4.2            py37h2bbff1b_1
typing_extensions         3.7.4.3            pyha847dfd_0
ujson                     4.0.2            py37hd77b12b_0
urllib3                   1.26.4             pyhd3eb1b0_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wcwidth                   0.2.5                      py_0
webencodings              0.5.1                    py37_1
wheel                     0.36.2             pyhd3eb1b0_0
widgetsnbextension        3.5.1                    py37_0
win_inet_pton             1.1.0            py37haa95532_0
wincertstore              0.2                      py37_0
winkerberos               0.7.0            py37he774522_1
winpty                    0.4.3                         4
wrapt                     1.12.1           py37he774522_1
x86cpu                    0.4                      py37_1    esri
xarray                    0.17.0             pyhd3eb1b0_0
xeus                      0.24.1                        1    esri
xlrd                      2.0.1              pyhd3eb1b0_0
xlwt                      1.3.0                    py37_0
xtl                       0.6.5                         1    esri
xz                        5.2.5                h62dcd97_0
yaml                      0.2.5                         0    esri
zeromq                    4.3.2                         2    esri
zipp                      3.4.1              pyhd3eb1b0_0
zlib                      1.2.11               h62dcd97_4
zstd                      1.4.5                h04227a9_0
Detteor commented 3 years ago

@UnderJollyRoger

Most of the time the script runs fine for me, but every once in while when trying to add to the Enterprise Portal it starts to have issues. The issues are not present when I submit anything to AGOL.

I chalked it up to my enterprise being at that age that things start to go wrong with the more modern tooling.

UnderJollyRoger commented 3 years ago

@Detteor Same. With AGOL I can add and publish consistently. With Enterprise it fails 9/10. I have a case going with an ESRI analyst. He found that the POST request to the /addItem endpoint of the Portal is being redirected.

Since that redirection is happening within the initial request there is an invalid response sent to the JSONDecode section of the code, hence the JSONDecodeError that's been so persistent.

I'll report back when I hear a resolution.

UnderJollyRoger commented 3 years ago

@achapkowski @Detteor After some additional help from an ESRI analyst I have a more or less stable script to publish a CSV to Portal as a feature service. And because it's a reoccurring process there's an additional script that overwrites the feature service. See this post.

achapkowski commented 3 years ago

@UnderJollyRoger thank you for the update