Event Streaming: Error while trying to Stream Child Table Updates to Consumer Doctype that has no Child Table

landersaga commented 7 months ago

Description of the issue

We have set up Event Streaming for 3 sites, 1 producer and 2 consumers. We mapped several ERPNext doctype with our own custom doctype. If user create document of a doctype that consisting child table with a row, and we mapped that doctype with the custom doctype that has no child table, it caused error because the stream trying to search for the child table Document Type Mapping on the consumer site even though we're not mapped the child table. This issue stops the entire event streaming, everytime the stream is triggered, it will caught at the same Event Update Log that caused this error and break the job.

Output of bench version

5.20.0

Steps to reproduce the issue

Create Event Streaming for streaming producer doctype that has child table to the consumer Doctype that has no child table
Set Document Type Mapping for the Doctypes (do not try to map the producer child table)
Create or update parent document on the producer site and edit the child table row

Observed result

Caught DoesNotExistsError

Expected result

Event Stream only stream field that has been set up on the Document Type Mapping

Stacktrace / full error message

Traceback with variables (most recent call last):
  File "apps/frappe/frappe/utils/background_jobs.py", line 194, in execute_job
    retval = method(**kwargs)
      site = 'crm.uniramagroup.co.id'
      method = <function pull_from_node at 0x7a85571ed6c0>
      event = None
      job_name = 'frappe.event_streaming.doctype.event_producer.event_producer.pull_from_node'
      kwargs = {'event_producer': 'https://erp.unirama.id'}
      user = 'udncrm.eventstream@example.com'
      is_async = True
      retry = 0
      retval = None
      method_name = 'frappe.event_streaming.doctype.event_producer.event_producer.pull_from_node'
      before_job_task = 'frappe.monitor.start'
  File "apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py", line 247, in pull_from_node
    update = get_mapped_update(update, producer_site)
      event_producer = <EventProducer: https://erp.unirama.id>
      producer_site = <frappe.frappeclient.FrappeClient object at 0x7a85570b2cb0>
      last_update = '2024-03-28 14:11:01.013222'
      doctypes = ['Address', 'Customer', 'Item']
      mapping_config = {'Customer': 'Customer ERP', 'Item': 'Item ERP'}
      naming_config = {'Address': 1, 'Customer': 1, 'Item': 1}
      updates = [{'update_type': 'Update', 'ref_doctype': 'Customer', 'docname': 'CS003431', 'data': '{\n "added": {\n  "credit_limits": [\n   {\n    "__unsaved": 1,\n    "bypass_credit_limit_check": 0,\n    "company": "Unirama Duta Niaga",\n    "creation": "2024-03-28 13:58:18.673289",\n    "credit_limit": 0.0,\n    "docstatus": 0,\n    "doctype": "Customer Credit Limit",\n    "idx": 1,\n    "modified": "2024-03-28 14:12:20.463697",\n    "modified_by": "hartono.unirama@gmail.com",\n    "name": "04f2a0180b",\n    "owner": "hartono.unirama@gmail.com",\n    "parent": "CS003431",\n    "parentfield": "credit_limits",\n    "parenttype": "Customer"\n   }\n  ]\n },\n "changed": {\n  "nik": "3309100711970001",\n  "npwp_name": "0",\n  "npwp_no": "0",\n  "payment_terms": "12 Hari",\n  "represents_company": ""\n },\n "removed": {},\n "row_changed": {}\n}', 'name': 498585, 'creation': '2024-03-28 14:12:20.553664', 'use_same_name': 1, 'mapping': 'Customer ERP'}, {'update_type': 'Update', 'ref_doctype': 'Customer',...
      update = {'update_type': 'Update', 'ref_doctype': 'Customer', 'docname': 'CS003431', 'data': '{\n "added": {\n  "credit_limits": [\n   {\n    "__unsaved": 1,\n    "bypass_credit_limit_check": 0,\n    "company": "Unirama Duta Niaga",\n    "creation": "2024-03-28 13:58:18.673289",\n    "credit_limit": 0.0,\n    "docstatus": 0,\n    "doctype": "Customer Credit Limit",\n    "idx": 1,\n    "modified": "2024-03-28 14:12:20.463697",\n    "modified_by": "hartono.unirama@gmail.com",\n    "name": "04f2a0180b",\n    "owner": "hartono.unirama@gmail.com",\n    "parent": "CS003431",\n    "parentfield": "credit_limits",\n    "parenttype": "Customer"\n   }\n  ]\n },\n "changed": {\n  "nik": "3309100711970001",\n  "npwp_name": "0",\n  "npwp_no": "0",\n  "payment_terms": "12 Hari",\n  "represents_company": ""\n },\n "removed": {},\n "row_changed": {}\n}', 'name': 498585, 'creation': '2024-03-28 14:12:20.553664', 'use_same_name': 1, 'mapping': 'Customer ERP'}
      mapping = 'Customer ERP'
  File "apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py", line 543, in get_mapped_update
    mapped_update = mapping.get_mapped_update(update, producer_site)
      update = {'update_type': 'Update', 'ref_doctype': 'Customer', 'docname': 'CS003431', 'data': '{\n "added": {\n  "credit_limits": [\n   {\n    "__unsaved": 1,\n    "bypass_credit_limit_check": 0,\n    "company": "Unirama Duta Niaga",\n    "creation": "2024-03-28 13:58:18.673289",\n    "credit_limit": 0.0,\n    "docstatus": 0,\n    "doctype": "Customer Credit Limit",\n    "idx": 1,\n    "modified": "2024-03-28 14:12:20.463697",\n    "modified_by": "hartono.unirama@gmail.com",\n    "name": "04f2a0180b",\n    "owner": "hartono.unirama@gmail.com",\n    "parent": "CS003431",\n    "parentfield": "credit_limits",\n    "parenttype": "Customer"\n   }\n  ]\n },\n "changed": {\n  "nik": "3309100711970001",\n  "npwp_name": "0",\n  "npwp_no": "0",\n  "payment_terms": "12 Hari",\n  "represents_company": ""\n },\n "removed": {},\n "row_changed": {}\n}', 'name': 498585, 'creation': '2024-03-28 14:12:20.553664', 'use_same_name': 1, 'mapping': 'Customer ERP'}
      producer_site = <frappe.frappeclient.FrappeClient object at 0x7a85570b2cb0>
      mapping = <DocumentTypeMapping: Customer ERP>
  File "apps/frappe/frappe/event_streaming/doctype/document_type_mapping/document_type_mapping.py", line 92, in get_mapped_update
    mapping = self.map_rows(update_diff, mapping, producer_site, operation="added")
      self = <DocumentTypeMapping: Customer ERP>
      update = {'update_type': 'Update', 'ref_doctype': 'Customer', 'docname': 'CS003431', 'data': '{\n "added": {\n  "credit_limits": [\n   {\n    "__unsaved": 1,\n    "bypass_credit_limit_check": 0,\n    "company": "Unirama Duta Niaga",\n    "creation": "2024-03-28 13:58:18.673289",\n    "credit_limit": 0.0,\n    "docstatus": 0,\n    "doctype": "Customer Credit Limit",\n    "idx": 1,\n    "modified": "2024-03-28 14:12:20.463697",\n    "modified_by": "hartono.unirama@gmail.com",\n    "name": "04f2a0180b",\n    "owner": "hartono.unirama@gmail.com",\n    "parent": "CS003431",\n    "parentfield": "credit_limits",\n    "parenttype": "Customer"\n   }\n  ]\n },\n "changed": {\n  "nik": "3309100711970001",\n  "npwp_name": "0",\n  "npwp_no": "0",\n  "payment_terms": "12 Hari",\n  "represents_company": ""\n },\n "removed": {},\n "row_changed": {}\n}', 'name': 498585, 'creation': '2024-03-28 14:12:20.553664', 'use_same_name': 1, 'mapping': 'Customer ERP'}
      producer_site = <frappe.frappeclient.FrappeClient object at 0x7a85570b2cb0>
      update_diff = {'added': {'credit_limits': [{'__unsaved': 1, 'bypass_credit_limit_check': 0, 'company': 'Unirama Duta Niaga', 'creation': '2024-03-28 13:58:18.673289', 'credit_limit': 0.0, 'docstatus': 0, 'doctype': 'Customer Credit Limit', 'idx': 1, 'modified': '2024-03-28 14:12:20.463697', 'modified_by': 'hartono.unirama@gmail.com', 'name': '04f2a0180b', 'owner': 'hartono.unirama@gmail.com', 'parent': 'CS003431', 'parentfield': 'credit_limits', 'parenttype': 'Customer'}]}, 'changed': {'default_payment_terms_template': '12 Hari', 'nik': '3309100711970001', 'npwp': '0', 'npwp_name': '0', 'represents_company': ''}, 'removed': {}, 'row_changed': {}}
      mapping = {'added': {'credit_limits': [{'__unsaved': 1, 'bypass_credit_limit_check': 0, 'company': 'Unirama Duta Niaga', 'creation': '2024-03-28 13:58:18.673289', 'credit_limit': 0.0, 'docstatus': 0, 'doctype': 'Customer Credit Limit', 'idx': 1, 'modified': '2024-03-28 14:12:20.463697', 'modified_by': 'hartono.unirama@gmail.com', 'name': '04f2a0180b', 'owner': 'hartono.unirama@gmail.com', 'parent': 'CS003431', 'parentfield': 'credit_limits', 'parenttype': 'Customer'}]}, 'changed': {'default_payment_terms_template': '12 Hari', 'nik': '3309100711970001', 'npwp': '0', 'npwp_name': '0', 'represents_company': ''}, 'removed': {}, 'row_changed': {}}
      dependencies = []
      doc_map = {'doc': '{\n "default_payment_terms_template": "12 Hari",\n "nik": "3309100711970001",\n "npwp": "0",\n "npwp_name": "0",\n "represents_company": ""\n}'}
      mapped_doc = '{\n "default_payment_terms_template": "12 Hari",\n "nik": "3309100711970001",\n "npwp": "0",\n "npwp_name": "0",\n "represents_company": ""\n}'
  File "apps/frappe/frappe/event_streaming/doctype/document_type_mapping/document_type_mapping.py", line 147, in map_rows
    table_map = frappe.get_doc("Document Type Mapping", table_map)
      self = <DocumentTypeMapping: Customer ERP>
      update_diff = {'added': {'credit_limits': [{'__unsaved': 1, 'bypass_credit_limit_check': 0, 'company': 'Unirama Duta Niaga', 'creation': '2024-03-28 13:58:18.673289', 'credit_limit': 0.0, 'docstatus': 0, 'doctype': 'Customer Credit Limit', 'idx': 1, 'modified': '2024-03-28 14:12:20.463697', 'modified_by': 'hartono.unirama@gmail.com', 'name': '04f2a0180b', 'owner': 'hartono.unirama@gmail.com', 'parent': 'CS003431', 'parentfield': 'credit_limits', 'parenttype': 'Customer'}]}, 'changed': {'default_payment_terms_template': '12 Hari', 'nik': '3309100711970001', 'npwp': '0', 'npwp_name': '0', 'represents_company': ''}, 'removed': {}, 'row_changed': {}}
      mapping = {'added': {'credit_limits': [{'__unsaved': 1, 'bypass_credit_limit_check': 0, 'company': 'Unirama Duta Niaga', 'creation': '2024-03-28 13:58:18.673289', 'credit_limit': 0.0, 'docstatus': 0, 'doctype': 'Customer Credit Limit', 'idx': 1, 'modified': '2024-03-28 14:12:20.463697', 'modified_by': 'hartono.unirama@gmail.com', 'name': '04f2a0180b', 'owner': 'hartono.unirama@gmail.com', 'parent': 'CS003431', 'parentfield': 'credit_limits', 'parenttype': 'Customer'}]}, 'changed': {'default_payment_terms_template': '12 Hari', 'nik': '3309100711970001', 'npwp': '0', 'npwp_name': '0', 'represents_company': ''}, 'removed': {}, 'row_changed': {}}
      producer_site = <frappe.frappeclient.FrappeClient object at 0x7a85570b2cb0>
      operation = 'added'
      remote_fields = []
      tablename = 'credit_limits'
      entries = [{'__unsaved': 1, 'bypass_credit_limit_check': 0, 'company': 'Unirama Duta Niaga', 'creation': '2024-03-28 13:58:18.673289', 'credit_limit': 0.0, 'docstatus': 0, 'doctype': 'Customer Credit Limit', 'idx': 1, 'modified': '2024-03-28 14:12:20.463697', 'modified_by': 'hartono.unirama@gmail.com', 'name': '04f2a0180b', 'owner': 'hartono.unirama@gmail.com', 'parent': 'CS003431', 'parentfield': 'credit_limits', 'parenttype': 'Customer'}]
      local_table_name = 'customer_finance'
      table_map = None
  File "apps/frappe/frappe/__init__.py", line 1197, in get_doc
    doc = frappe.model.document.get_doc(*args, **kwargs)
      args = ('Document Type Mapping', None)
      kwargs = {}
      frappe = <module 'frappe' from 'apps/frappe/frappe/__init__.py'>
  File "apps/frappe/frappe/model/document.py", line 75, in get_doc
    return controller(*args, **kwargs)
      args = ('Document Type Mapping', None)
      kwargs = {}
      doctype = 'Document Type Mapping'
      controller = <class 'frappe.event_streaming.doctype.document_type_mapping.document_type_mapping.DocumentTypeMapping'>
  File "apps/frappe/frappe/model/document.py", line 107, in __init__
    self.load_from_db()
      self = <DocumentTypeMapping: unsaved>
      args = ('Document Type Mapping', None)
      kwargs = {}
      __class__ = <class 'frappe.model.document.Document'>
  File "apps/frappe/frappe/model/document.py", line 150, in load_from_db
    frappe.throw(
      self = <DocumentTypeMapping: unsaved>
      d = None
      __class__ = <class 'frappe.model.document.Document'>
  File "apps/frappe/frappe/__init__.py", line 540, in throw
    msgprint(
      msg = 'Document Type Mapping None not found'
      exc = <class 'frappe.exceptions.DoesNotExistError'>
      title = None
      is_minimizable = False
      wide = False
      as_list = False
  File "apps/frappe/frappe/__init__.py", line 508, in msgprint
    _raise_exception()
      title = None
      as_table = False
      as_list = False
      indicator = 'red'
      alert = False
      primary_action = None
      is_minimizable = False
      wide = False
      realtime = False
      sys = <module 'sys' (built-in)>
      out = {'message': 'Document Type Mapping None not found', 'title': 'Message', 'indicator': 'red', 'raise_exception': 1}
      _raise_exception = <function msgprint.<locals>._raise_exception at 0x7a8556e237f0>
      _strip_html_tags = <functools._lru_cache_wrapper object at 0x7a8556e90670>
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'Document Type Mapping None not found'
      raise_exception = <class 'frappe.exceptions.DoesNotExistError'>
      strip_html_tags = <function strip_html_tags at 0x7a855d492a70>
  File "apps/frappe/frappe/__init__.py", line 454, in _raise_exception
    raise raise_exception(msg)
      inspect = <module 'inspect' from '/usr/lib/python3.10/inspect.py'>
      msg = 'Document Type Mapping None not found'
      raise_exception = <class 'frappe.exceptions.DoesNotExistError'>
frappe.exceptions.DoesNotExistError: Document Type Mapping None not found

Additional information

Frappe Version: v14.62.4

karanwilson commented 7 months ago

Hi @landersaga, is this issue currently unresolved?

karanwilson commented 7 months ago

Hi @landersaga, Quoting from your comment:- "If user create document of a doctype that consisting child table with a row, and we mapped that doctype with the custom doctype that has no child table..."

You need to create a new 'Mapping Configuration' for the child-table of the doctype you are streaming - in addition to the 'Mapping Configuration' of the parent doctype.

Refer:- https://docs.erpnext.com/docs/user/manual/en/event_streaming#3-6-3-mapping-for-doctypes-having-child-tables

@nabinhait

landersaga commented 6 months ago

Hi @landersaga, is this issue currently unresolved?

Yes, this issue currently unresolved

landersaga commented 6 months ago

Hi @landersaga, Quoting from your comment:- "If user create document of a doctype that consisting child table with a row, and we mapped that doctype with the custom doctype that has no child table..."

You need to create a new 'Mapping Configuration' for the child-table of the doctype you are streaming - in addition to the 'Mapping Configuration' of the parent doctype.

Refer:- https://docs.erpnext.com/docs/user/manual/en/event_streaming#3-6-3-mapping-for-doctypes-having-child-tables

@nabinhait

There is difference table structure on both sides. The one on the producer trying to stream updated row of child table, while the consumer doctype doesn't consisting that child table

frappe / event_streaming