marcosmarxm
commented
2 months ago @Samuel-Dittmann are you able to run tests and read command when developing the connector? Can you share the YAML file?
Open Samuel-Dittmann opened 3 months ago
marcosmarxm
commented
2 months ago @Samuel-Dittmann are you able to run tests and read command when developing the connector? Can you share the YAML file?
Samuel-Dittmann
commented
2 months ago Hey @marcosmarxm the tests run fine and display the data that I want to see even when Im using OAuth2.
Its just when I publish it and try to use it in an active connection when it does not work. Also here is the yaml which I shrinked down and removed my tenant-id to only show one stream:
spec:
type: Spec
connection_specification:
type: object
$schema: http://json-schema.org/draft-07/schema#
required:
- client_id
- client_secret
properties:
client_id:
type: string
order: 0
title: Client ID
airbyte_secret: true
client_secret:
type: string
order: 1
title: Client secret
airbyte_secret: true
additionalProperties: true
type: DeclarativeSource
check:
type: CheckStream
stream_names:
- sharepoint_users
streams:
- name: sharepoint_users
type: DeclarativeStream
retriever:
type: SimpleRetriever
paginator:
type: NoPagination
requester:
path: sites/root/lists('Benutzerinformationsliste')/items?expand=fields
type: HttpRequester
url_base: https://graph.microsoft.com/v1.0/
http_method: GET
authenticator:
type: OAuthAuthenticator
scopes: []
client_id: '{{ config[''client_id''] }}'
grant_type: client_credentials
client_secret: '{{ config[''client_secret''] }}'
refresh_request_body:
scope: https://graph.microsoft.com/.default
token_refresh_endpoint: >-
https://login.microsoftonline.com/<my-tenant-id>/oauth2/v2.0/token
request_headers: {}
request_body_json: {}
request_parameters: {}
record_selector:
type: RecordSelector
extractor:
type: DpathExtractor
field_path:
- value
- '*'
- fields
partition_router: []
primary_key:
- id
schema_loader:
type: InlineSchemaLoader
schema:
type: object
$schema: http://json-schema.org/schema#
properties:
id:
type: string
Edit:
type: string
Name:
type: string
EMail:
type: string
Notes:
type: string
Title:
type: string
Office:
type: string
Created:
type: string
Deleted:
type: boolean
ImnName:
type: string
Picture:
type: object
properties:
Url:
type: string
Description:
type: string
EditUser:
type: string
JobTitle:
type: string
LastName:
type: string
Modified:
type: string
UserName:
type: string
FirstName:
type: string
LinkTitle:
type: string
WorkPhone:
type: string
Department:
type: string
SipAddress:
type: string
'@odata.etag':
type: string
Attachments:
type: boolean
ContentType:
type: string
IsSiteAdmin:
type: boolean
MobilePhone:
type: string
UserSelection:
type: string
AuthorLookupId:
type: string
EditorLookupId:
type: string
ItemChildCount:
type: string
UserInfoHidden:
type: boolean
_ComplianceTag:
type: string
ContentTypeDisp:
type: string
LinkTitleNoMenu:
type: string
FolderChildCount:
type: string
_ComplianceFlags:
type: string
_UIVersionString:
type: string
AppAuthorLookupId:
type: string
AppEditorLookupId:
type: string
SPSResponsibility:
type: string
SPSPictureTimestamp:
type: string
_ComplianceTagUserId:
type: string
PictureOnly_Size_36px:
type: string
PictureOnly_Size_48px:
type: string
PictureOnly_Size_72px:
type: string
NameWithPictureAndDetails:
type: string
_ComplianceTagWrittenTime:
type: string
SPSPicturePlaceholderState:
type: number
SPSPictureExchangeSyncState:
type: number
version: 0.65.0
metadata:
autoImportSchema:
countries: true
locations: true
delegation: true
cost_centers: true
company_brands: true
legal_entities: true
country_regions: true
sharepoint_users: true
cost_center_types: true
delegation_data_scopes: true
legal_entity_categories: true
division_organisational_elements: truePlease note, that I have already tried to delete all streams except for one - this did not solve the problem either. Thats why I think it could be an authentication issue within Airbyte.
Connector Name
source-custom_connector-sharepoint_lists
Connector Version
12
What step the error happened?
Configuring a new connector
Relevant information
Hey,
Im working on a custom connector using the UI builder to get data from a few of our SharePoint Online Lists into Databricks. I have configured each list as its own stream and authenticated using an Entra ID App Registration. In the builder everything works perfectly. As soon as I publish it and want to create a connection, I get the following error:
Configuration check failed
Unable to connect to stream cost_centers - HTTPSConnectionPool(host='[login.microsoftonline.com](http://login.microsoftonline.com/)', port=443): Max retries exceeded with url: /my-tenant-id/oauth2/v2.0/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd22c6f54f0>: Failed to establish a new connection: [Errno 101] Network unreachable'))However I will not get the error when I obtain the Bearer Token externaly via Postman and switch the connector to Bearer Type auth - then everything works as expected aswell.
Relevant log output
Contribute