Open tulequ opened 1 year ago
Can you provide the complete ApisixUpstream configuration? And is it referenced by any ApisixRoute resources? Does it include the service that it contains?
Moreover, has the CRD used by APISIX Ingress been updated?
@tulequ Any updates here?
We have this error when trying to use this with a translated Ingress.
The docs make it seem like adding an ApisixUpstream is transparent, by simply having the same name as the service. The error message is simply not very helpful, but very persistent.
2024-01-09T22:03:39+08:00 error apisix/apisix_upstream.go:332 failed to get upstream kunde--production_paco-cs_80_service: not found
2024-01-09T22:03:39+08:00 warn apisix/apisix_upstream.go:488 sync ApisixUpstream failed, will retry {"object": {"Type":1,"Object":{"Key":"kunde--production/paco-cs","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
2024-01-09T22:04:39+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "a4f87d6b", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a4f87d6b", "cluster": "default"}
2024-01-09T22:04:39+08:00 error apisix/apisix_upstream.go:332 failed to get upstream kunde--production_paco-cs_80_service: not found
2024-01-09T22:04:39+08:00 warn apisix/apisix_upstream.go:488 sync ApisixUpstream failed, will retry {"object": {"Type":1,"Object":{"Key":"kunde--production/paco-cs","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
2024-01-09T22:05:39+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "a4f87d6b", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a4f87d6b", "cluster": "default"}
2024-01-09T22:05:39+08:00 error apisix/apisix_upstream.go:332 failed to get upstream kunde--production_paco-cs_80_service: not found
2024-01-09T22:05:39+08:00 warn apisix/apisix_upstream.go:488 sync ApisixUpstream failed, will retry {"object": {"Type":1,"Object":{"Key":"kunde--production/paco-cs","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
the events have a few more hints, since they reference ApisixIngress, but it's not great either:
ApisixIngress synced failed, with error: not found
Source: ApisixIngress
Count: 11
Sub-object:
Last seen: 2024-01-09T14:54:41+01:00
I do have the same problem. I added a route which is linked to Upstream to reach an external service, basically like described here https://apisix.apache.org/docs/ingress-controller/tutorials/external-service/, but it doesn't work.
In the Ingress Controller I do get the following errors
2024-02-14T15:08:40+08:00 error apisix/apisix_route.go:379 failed to sync ApisixRoute to apisix {"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [53af2afe], response code: 404\"}\n\n\n"}
2024-02-14T15:08:40+08:00 warn apisix/apisix_route.go:481 sync ApisixRoute failed, will retry {"object": {"Type":4,"Object":{"Key":"tracs-dev/tracs-dev-notifications-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [53af2afe], response code: 404\"}\n\n\n"}
2024-02-14T15:08:40+08:00 warn apisix/cluster.go:1164 upstream not found {"id": "53af2afe", "url": "http://tracs-dev-apisix-admin.tracs-dev.svc.cluster.local:9180/apisix/admin/upstreams/53af2afe", "cluster": "default"}
2024-02-14T15:08:40+08:00 error apisix/apisix_upstream.go:375 failed to get upstream tracs-dev_tracs-dev-notification-upstream: not found
2024-02-14T15:08:40+08:00 warn apisix/apisix_upstream.go:488 sync ApisixUpstream failed, will retry {"object": {"Type":4,"Object":{"Key":"tracs-dev/tracs-dev-notification-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
2024-02-14T15:08:40+08:00 error translation/apisix_route.go:236 failed to translate ApisixUpstream at Upstream[0] {"error": "ApisixUpstream tracs-dev/tracs-dev-notification-upstream ExternalNodes[0]'s name as Domain must match lowercase RFC 1123 subdomain. a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character", "apisix_upstream": "tracs-dev/tracs-dev-notification-upstream"}
2024-02-14T15:08:40+08:00 error apisix/route.go:138 failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [53af2afe], response code: 404"}
and this is the status of the Upstream
Status:
Conditions:
Message: not found
Observed Generation: 3
Reason: ResourceSyncAborted
Status: False
Type: ResourcesAvailable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ResourceSyncAborted 66s (x25052 over 18h) ApisixIngress ApisixIngress synced failed, with error: not found
I made one oberservation, in the error logs the ingress is looking for tracs-dev_tracs-dev-notification-upstream
but the upstream is just called 'tracs-dev-notification-upstream' and is defined in the namespace tracs-dev
Used versions: tracs-dev-apisix-745b4b89b6 | OK | apisix: apache/apisix:3.5.0-debian tracs-dev-ingress-controller-7fcd8d7bf8 | OK | ingress-controller: apache/apisix-ingress-controller:1.7.0
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
Any update on this problem? I am facing the same issue.
I am facing the same issue. apache/apisix-ingress-controller:1.7.1
CustomResourceDefinition:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: apisixupstreams.apisix.apache.org
uid: dc3b7946-aa5d-47d9-8572-52234a053a80
resourceVersion: '284807486'
generation: 2
creationTimestamp: '2023-09-06T07:46:42Z'
labels:
k8slens-edit-resource-version: v1
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"apisixupstreams.apisix.apache.org"},"spec":{"group":"apisix.apache.org","names":{"kind":"ApisixUpstream","plural":"apisixupstreams","shortNames":["au"],"singular":"apisixupstream"},"scope":"Namespaced","versions":[{"name":"v2","schema":{"openAPIV3Schema":{"properties":{"spec":{"properties":{"discovery":{"description":"Discovery
is used to configure service discovery for
upstream","properties":{"args":{"type":"object","x-kubernetes-preserve-unknown-fields":true},"serviceName":{"type":"string"},"type":{"type":"string"}},"type":"object"},"externalNodes":{"description":"ExternalNodes
contains external nodes the Upstream should use If this field is set, the
upstream will use these nodes directly without any further
resolves","items":{"description":"ApisixUpstreamExternalNode is the
external node
conf","properties":{"name":{"type":"string"},"port":{"type":"integer"},"type":{"type":"string"},"weight":{"type":"integer"}},"type":"object"},"type":"array"},"healthCheck":{"anyOf":[{"required":["active"]},{"required":["active","passive"]}],"properties":{"active":{"properties":{"concurrency":{"minimum":1,"type":"integer"},"healthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"interval":{"type":"string"},"successes":{"maximum":254,"minimum":1,"type":"integer"}},"type":"object"},"host":{"pattern":"^\\*?[0-9a-zA-Z-._]+$","type":"string"},"httpPath":{"minLength":1,"type":"string"},"port":{"maximum":65535,"minimum":1,"type":"integer"},"requestHeaders":{"items":{"type":"string"},"minItems":1,"type":"array"},"strictTLS":{"type":"boolean"},"timeout":{"minimum":0,"type":"number"},"type":{"enum":["http","https","tcp"],"type":"string"},"unhealthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"httpFailures":{"maximum":254,"minimum":1,"type":"integer"},"interval":{"type":"string"},"tcpFailures":{"maximum":254,"minimum":1,"type":"integer"},"timeouts":{"minimum":0,"type":"integer"}},"type":"object"}},"type":"object"},"passive":{"properties":{"healthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"successes":{"maximum":254,"minimum":1,"type":"integer"}},"type":"object"},"type":{"enum":["http","https","tcp"],"type":"string"},"unhealthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"httpFailures":{"maximum":254,"minimum":1,"type":"integer"},"tcpFailures":{"maximum":254,"minimum":1,"type":"integer"},"timeouts":{"minimum":0,"type":"integer"}},"type":"object"}},"type":"object"}},"type":"object"},"ingressClassName":{"type":"string"},"loadbalancer":{"properties":{"hashOn":{"enum":["vars","vars_combinations","header","cookie","consumer"],"type":"string"},"key":{"type":"string"},"type":{"enum":["roundrobin","chash","ewma","least_conn"],"type":"string"}},"required":["type"],"type":"object"},"passHost":{"enum":["pass","node","rewrite"],"type":"string"},"portLevelSettings":{"items":{"properties":{"healthCheck":{"anyOf":[{"required":["active"]},{"required":["active","passive"]}],"properties":{"active":{"properties":{"concurrency":{"minimum":1,"type":"integer"},"healthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"interval":{"type":"string"},"successes":{"maximum":254,"minimum":1,"type":"integer"}},"type":"object"},"host":{"pattern":"^\\*?[0-9a-zA-Z-._]+$","type":"string"},"httpPath":{"minLength":1,"type":"string"},"port":{"maximum":65535,"minimum":1,"type":"integer"},"requestHeaders":{"items":{"type":"string"},"minItems":1,"type":"array"},"strictTLS":{"type":"boolean"},"timeout":{"minimum":0,"type":"number"},"type":{"enum":["http","https","tcp"],"type":"string"},"unhealthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"httpFailures":{"maximum":254,"minimum":1,"type":"integer"},"interval":{"type":"string"},"tcpFailures":{"maximum":254,"minimum":1,"type":"integer"},"timeout":{"type":"string"}},"type":"object"}},"type":"object"},"passive":{"properties":{"healthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"successes":{"maximum":254,"minimum":1,"type":"integer"}},"type":"object"},"type":{"enum":["http","https","tcp"],"type":"string"},"unhealthy":{"properties":{"httpCodes":{"items":{"maximum":599,"minimum":200,"type":"integer"},"minItems":1,"type":"array"},"httpFailures":{"maximum":254,"minimum":1,"type":"integer"},"tcpFailures":{"maximum":254,"minimum":1,"type":"integer"},"timeout":{"type":"string"}},"type":"object"}},"type":"object"}},"type":"object"},"loadbalancer":{"properties":{"hashOn":{"enum":["vars","vars_combinations","header","cookie","consumer"],"type":"string"},"key":{"type":"string"},"type":{"enum":["roundrobin","chash","ewma","least_conn"],"type":"string"}},"required":["type"],"type":"object"},"port":{"maximum":65535,"minimum":1,"type":"integer"},"retries":{"minimum":0,"type":"integer"},"scheme":{"enum":["http","grpc"],"type":"string"},"timeout":{"properties":{"connect":{"type":"string"},"read":{"type":"string"},"send":{"type":"string"}},"type":"object"}},"type":"object"},"type":"array"},"retries":{"minimum":0,"type":"integer"},"scheme":{"enum":["http","grpc","https","grpcs"],"type":"string"},"subsets":{"items":{"properties":{"labels":{"type":"object","x-kubernetes-preserve-unknown-fields":true},"name":{"minLength":1,"type":"string"}},"required":["name","labels"],"type":"object"},"type":"array"},"timeout":{"properties":{"connect":{"type":"string"},"read":{"type":"string"},"send":{"type":"string"}},"type":"object"},"tlsSecret":{"description":"ApisixSecret
describes the Kubernetes Secret name and
namespace.","properties":{"name":{"minLength":1,"type":"string"},"namespace":{"minLength":1,"type":"string"}},"required":["name","namespace"],"type":"object"},"upstreamHost":{"pattern":"^\\*?[0-9a-zA-Z-._]+$","type":"string"}},"type":"object"},"status":{"properties":{"conditions":{"items":{"properties":{"message":{"type":"string"},"observedGeneration":{"type":"integer"},"reason":{"type":"string"},"status":{"type":"string"},"type":{"type":"string"}},"type":"object"},"type":"array"}},"type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]}}
managedFields:
- manager: Go-http-client
operation: Update
apiVersion: apiextensions.k8s.io/v1
time: '2023-09-06T07:46:42Z'
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:conversion:
.: {}
f:strategy: {}
f:group: {}
f:names:
f:kind: {}
f:listKind: {}
f:plural: {}
f:shortNames: {}
f:singular: {}
f:scope: {}
- manager: kube-apiserver
operation: Update
apiVersion: apiextensions.k8s.io/v1
time: '2023-09-06T07:46:47Z'
fieldsType: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:kind: {}
f:listKind: {}
f:plural: {}
f:shortNames: {}
f:singular: {}
f:conditions:
k:{"type":"Established"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamesAccepted"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
subresource: status
- manager: kubectl-client-side-apply
operation: Update
apiVersion: apiextensions.k8s.io/v1
time: '2024-06-28T02:39:11Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
f:versions: {}
- manager: node-fetch
operation: Update
apiVersion: apiextensions.k8s.io/v1
time: '2024-06-28T02:42:56Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:k8slens-edit-resource-version: {}
selfLink: >-
/apis/apiextensions.k8s.io/v1/customresourcedefinitions/apisixupstreams.apisix.apache.org
status:
conditions:
- type: NamesAccepted
status: 'True'
lastTransitionTime: '2023-09-06T07:46:42Z'
reason: NoConflicts
message: no conflicts found
- type: Established
status: 'True'
lastTransitionTime: '2023-09-06T07:46:47Z'
reason: InitialNamesAccepted
message: the initial names have been accepted
acceptedNames:
plural: apisixupstreams
singular: apisixupstream
shortNames:
- au
kind: ApisixUpstream
listKind: ApisixUpstreamList
storedVersions:
- v2
spec:
group: apisix.apache.org
names:
plural: apisixupstreams
singular: apisixupstream
shortNames:
- au
kind: ApisixUpstream
listKind: ApisixUpstreamList
scope: Namespaced
versions:
- name: v2
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
discovery:
description: >-
Discovery is used to configure service discovery for
upstream
type: object
properties:
args:
type: object
x-kubernetes-preserve-unknown-fields: true
serviceName:
type: string
type:
type: string
externalNodes:
description: >-
ExternalNodes contains external nodes the Upstream should
use If this field is set, the upstream will use these nodes
directly without any further resolves
type: array
items:
description: ApisixUpstreamExternalNode is the external node conf
type: object
properties:
name:
type: string
port:
type: integer
type:
type: string
weight:
type: integer
healthCheck:
type: object
anyOf:
- required:
- active
- required:
- active
- passive
properties:
active:
type: object
properties:
concurrency:
type: integer
minimum: 1
healthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
interval:
type: string
successes:
type: integer
maximum: 254
minimum: 1
host:
type: string
pattern: ^\*?[0-9a-zA-Z-._]+$
httpPath:
type: string
minLength: 1
port:
type: integer
maximum: 65535
minimum: 1
requestHeaders:
type: array
minItems: 1
items:
type: string
strictTLS:
type: boolean
timeout:
type: number
minimum: 0
type:
type: string
enum:
- http
- https
- tcp
unhealthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
httpFailures:
type: integer
maximum: 254
minimum: 1
interval:
type: string
tcpFailures:
type: integer
maximum: 254
minimum: 1
timeouts:
type: integer
minimum: 0
passive:
type: object
properties:
healthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
successes:
type: integer
maximum: 254
minimum: 1
type:
type: string
enum:
- http
- https
- tcp
unhealthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
httpFailures:
type: integer
maximum: 254
minimum: 1
tcpFailures:
type: integer
maximum: 254
minimum: 1
timeouts:
type: integer
minimum: 0
ingressClassName:
type: string
loadbalancer:
type: object
required:
- type
properties:
hashOn:
type: string
enum:
- vars
- vars_combinations
- header
- cookie
- consumer
key:
type: string
type:
type: string
enum:
- roundrobin
- chash
- ewma
- least_conn
passHost:
type: string
enum:
- pass
- node
- rewrite
portLevelSettings:
type: array
items:
type: object
properties:
healthCheck:
type: object
anyOf:
- required:
- active
- required:
- active
- passive
properties:
active:
type: object
properties:
concurrency:
type: integer
minimum: 1
healthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
interval:
type: string
successes:
type: integer
maximum: 254
minimum: 1
host:
type: string
pattern: ^\*?[0-9a-zA-Z-._]+$
httpPath:
type: string
minLength: 1
port:
type: integer
maximum: 65535
minimum: 1
requestHeaders:
type: array
minItems: 1
items:
type: string
strictTLS:
type: boolean
timeout:
type: number
minimum: 0
type:
type: string
enum:
- http
- https
- tcp
unhealthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
httpFailures:
type: integer
maximum: 254
minimum: 1
interval:
type: string
tcpFailures:
type: integer
maximum: 254
minimum: 1
timeout:
type: string
passive:
type: object
properties:
healthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
successes:
type: integer
maximum: 254
minimum: 1
type:
type: string
enum:
- http
- https
- tcp
unhealthy:
type: object
properties:
httpCodes:
type: array
minItems: 1
items:
type: integer
maximum: 599
minimum: 200
httpFailures:
type: integer
maximum: 254
minimum: 1
tcpFailures:
type: integer
maximum: 254
minimum: 1
timeout:
type: string
loadbalancer:
type: object
required:
- type
properties:
hashOn:
type: string
enum:
- vars
- vars_combinations
- header
- cookie
- consumer
key:
type: string
type:
type: string
enum:
- roundrobin
- chash
- ewma
- least_conn
port:
type: integer
maximum: 65535
minimum: 1
retries:
type: integer
minimum: 0
scheme:
type: string
enum:
- http
- grpc
timeout:
type: object
properties:
connect:
type: string
read:
type: string
send:
type: string
retries:
type: integer
minimum: 0
scheme:
type: string
enum:
- http
- grpc
- https
- grpcs
subsets:
type: array
items:
type: object
required:
- name
- labels
properties:
labels:
type: object
x-kubernetes-preserve-unknown-fields: true
name:
type: string
minLength: 1
timeout:
type: object
properties:
connect:
type: string
read:
type: string
send:
type: string
tlsSecret:
description: >-
ApisixSecret describes the Kubernetes Secret name and
namespace.
type: object
required:
- name
- namespace
properties:
name:
type: string
minLength: 1
namespace:
type: string
minLength: 1
upstreamHost:
type: string
pattern: ^\*?[0-9a-zA-Z-._]+$
status:
type: object
properties:
conditions:
type: array
items:
type: object
properties:
message:
type: string
observedGeneration:
type: integer
reason:
type: string
status:
type: string
type:
type: string
subresources:
status: {}
conversion:
strategy: None
version 1.8.0(latest) the same issue on AKS : ApisixIngress synced failed, with error: not found
I'm also having the same issue. when will this be solved?
Current Behavior
When use ApisixUpstream with apisix ingress controller 1.7
It show this status:
event log:
But when I change image to 1.6.1, it was successful
If I change back to 1.7.0, it show not found again
Expected Behavior
It should be successful like before version
Error Logs
Steps to Reproduce
Environment
Docker image: apache/apisix-ingress-controller:1.7.0
Version: 1.7.0 Git SHA: no-git-module Go Version: go1.20.8 Building OS/Arch: linux/amd64 Running OS/Arch: linux/amd64