search

lavaloon-eg / ksa_compliance

KSA Compliance App for KSA E-invoice
https://www.lavaloon.com/
GNU Affero General Public License v3.0
34 stars 23 forks source link

Integration error #62

Closed saud-2 closed 3 months ago

saud-2 commented 3 months ago

image

I completed the steps of integration with no issue until the last step ( get production CSID) I got the attached Error.

Regards

mhaggag commented 3 months ago

Hello,

We've previously tested against the sandbox environment and went live for customers on the production environment. We'll check whether there are issues on the simulation environment.

Do you get error details in the Error Log list? If you have SSH access, you can also look for logs/zatca.log inside the bench. It may contain more details.

saud-2 commented 3 months ago

No details in logs/zatca.log

2024-06-20 09:54:48,032 INFO zatca CSR config: csr.common.name=XXXXXXX csr.serial.number= 1-ERPNext|2-15|3-1 csr.organization.identifier=XXXXXXXXX csr.organization.unit.name=XXXXXXX csr.organization.name=XXXXXXXX csr.country.name=SA csr.invoice.type=1100 csr.location.address=XXXXXXXXXXX csr.industry.business.category=Retail 2024-06-20 09:54:48,047 INFO zatca Running: ['/home/frappe/frappe-bench/sites/zatca/zatca-cli/bin/zatca-cli', 'csr', '-c', '/tmp/tmp_802swor-csr-XXXXXXXXX.properties', '-o', 'XXXXXXXXXX.csr', '-k', 'XXXXXXXXX.privkey'] 2024-06-20 09:54:49,352 INFO zatca CSR generated successfully 2024-06-20 09:55:53,448 ERROR zatca An HTTP error occurred 2024-06-20 09:55:53,453 INFO zatca Response: {'code': 'Invalid-Request', 'message': 'System failed to process your request'}

saud-2 commented 3 months ago

even with sandbox environment . integration is done successfully however when I created new invoice become rejected and the reason was "requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://gw-fatoora.zatca.gov.sa/e-invoicing/developer-portal/invoices/reporting/single" and no error details in error log or zatca.log Note: OTP is from production portal there is no OTP for sandbox.

mhaggag commented 3 months ago
saud-2 commented 3 months ago

[XSD] validation result : PASSED [EN] validation result : PASSED [KSA] validation result : PASSED [QR] validation result : PASSED [SIGNATURE] validation result : PASSED [PIH] validation result : PASSED *** GLOBAL VALIDATION RESULT = PASSED

VAT registration number 3XXXXXXXXX00003

saud-2 commented 3 months ago

Compliance Check Result image

The result is accepted but in the same time there is error log.

image

Is there any error here?

mhaggag commented 3 months ago

There's no error in the compliance checks you're getting. They look fine.

The compliance check creates some invoices to submit to the compliance API then rolls them back. It uses the Error Log table to store the responses in all cases, success or failure, because the tabError Log table uses the MyISAM non-transactional engine, so the logs are stored even after roll-back.

Validation looks fine too. If these are results from the sandbox API, then the problem causing 400 on invoice reporting would likely be that the VAT you're using is not the hard-coded VAT they use in the sandbox.

saud-2 commented 3 months ago

The VAT I'm using is 3XXXXXXXXX00003 . Is it applicable with sandbox ?

mhaggag commented 3 months ago

I don't think so. Please try 399999999900003

من: Saud @.> ‏‏تم الإرسال: Monday, June 24, 2024 1:10:07 AM إلى: lavaloon-eg/ksa_compliance @.> نسخة: Muhammad Haggag @.>; Comment @.> ‏‏الموضوع: Re: [lavaloon-eg/ksa_compliance] Integration error (Issue #62)

The VAT I'm using is 3XXXXXXXXX00003 . Is it applicable with sandbox ?

— Reply to this email directly, view it on GitHubhttps://github.com/lavaloon-eg/ksa_compliance/issues/62#issuecomment-2185338465, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AALG35G2SWSYBBIQSZ7AQXDZI5BT7AVCNFSM6AAAAABJRRB3UOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGMZTQNBWGU. You are receiving this because you commented.Message ID: @.***>

saud-2 commented 3 months ago

I tried this one 399999999900003 and it is accepted but mine not and it is same code 3xxxxxxxxx00003. how to fix it ?

mhaggag commented 3 months ago

The sandbox is mainly used to make sure your invoices and configuration is supported by the app; i.e. it leads to valid invoices accepted by ZATCA. Thus, it doesn't really allow you to specify your own VAT and test with it. That's what the simulation environment is for.

If the invoice configurations you produce are limited, you can verify them all on the sandbox and then go live (production).

In the mean time, I will consult our quality team to see if they've checked the simulation environment and get back to you.

mhaggag commented 3 months ago

There's a fix on the development branch (develop) for the simulation environment if you'd like to give it a try. It's in version 0.20.1, but also relies on CLI version 2.0.1. If you want to give it a try:

saud-2 commented 3 months ago

I don't know how to switch to develop. shall I uninstall master branch first then install develop branch ?

mhaggag commented 3 months ago

It depends on whether this is self-hosted or on frappe cloud. On frappe cloud, you go to the bench, switch to the "Apps" tab, click the "..." next to the app and select "Change Branch".

Make sure you're using a test instance or site, since the develop branch is for stuff that's under test. If you're not in a hurry, you can wait until next week when it'll be merged into master following testing and any fixes necessary

saud-2 commented 3 months ago

installed develop branch and in last step " Get Production CSID" I got attached error.

image This is zatca.log

2024-06-28 11:45:32,914 INFO zatca Running: ['/home/frappe/frappe-bench/sites/zatca/zatca-cli/bin/zatca-cli', 'csr', '-c', '/tmp/tmpbmgvgji0-csr-310xxxxxxx00003.properties', '-o', '310xxxxxxx00003.csr', '-k', '310xxxxxxx00003.privkey'] 2024-06-28 11:45:33,492 INFO zatca CSR generated successfully 2024-06-28 11:46:28,784 ERROR zatca An HTTP error occurred: System failed to process your request 2024-06-28 11:46:28,784 INFO zatca Response: {"code":"Invalid-Request","message":"System failed to process your request"} 2024-06-28 11:47:55,281 ERROR zatca An HTTP error occurred: System failed to process your request 2024-06-28 11:47:55,282 INFO zatca Response: {"code":"Invalid-Request","message":"System failed to process your request"}

mhaggag commented 3 months ago

Is this on sandbox, simulation, or production environment? (developer-portal, simulation, or core in the URL). For the simulation environment the CSR command should include a -s flag to generate a simulation CSR, which doesn't seem to be the case here.

saud-2 commented 3 months ago

simulation. URL is https://gw-fatoora.zatca.gov.sa/e-invoicing/simulation/ OTP is from simulation portal . I don't type any command, I just put OTP.

mhaggag commented 3 months ago

Can you confirm that it's version 0.20.1? (from the help menu)

Perhaps we missed a step in the update on frappe cloud. I asked one of our frappe cloud admins and he said changing the branch on the bench should be followed up by updating the site itself. I'm not sure how to do that since I don't use frappe cloud much, I tend to work more with the self-hosted instances for our larger customers that don't fit on frappe cloud.

saud-2 commented 3 months ago

yes it's version is 0.20.1. and I am on self hosting not frappe cloud.

image

mhaggag commented 3 months ago

OK, good to know. Let's figure out why it's not detecting the simulation environment.

The code to detect the simulation environment is in apps/ksa_compliance/ksa_compliance/ksa_compliance/doctype/zatca_business_settings/zatca_business_settings.py. Open the file with an editor (vim or nano) and look for is_simulation_environment. It should be a property returning a boolean.

Next, we should try it out from python. Run bench console, then run:

settings = frappe.get_last_doc('ZATCA Business Settings')
settings.is_simulation_server

(You can replace get_last_doc with get_doc('ZATCA Business Settings', 'name-of-your-settings') if it's not the last document)

Does it properly report True for your environment?

saud-2 commented 3 months ago

apps/ksa_compliance/ksa_compliance/ksa_compliance/doctype/zatca_business_settings/zatca_business_settings.py. Open the file with an editor (vim or nano) and look for is_simulation_environment. It should be a property returning a boolean.

@property def is_simulation_server(self) -> bool: return self.fatoora_server_url.startswith('https://gw-fatoora.zatca.gov.sa/e-invoicing/simulation')

settings = frappe.get_last_doc('ZATCA Business Settings')

-bash: syntax error near unexpected token `('

mhaggag commented 3 months ago

settings = frappe.get_last_doc('ZATCA Business Settings')

-bash: syntax error near unexpected token `('

You need to paste those commands into a python console run with bench console. It should look something like this:

‏لقطة الشاشة 2024-06-28 في 7 45 47 م
saud-2 commented 3 months ago

:~/frappe-bench/apps/ksa_compliance/ksa_compliance/ksa_compliance/doctype/zatca_business_settings$ bench console Apps in this namespace: frappe, erpnext

In [1]: settings = frappe.get_last_doc('ZATCA Business Settings')

ProgrammingError Traceback (most recent call last) Cell In[1], line 1 ----> 1 settings = frappe.get_last_doc('ZATCA Business Settings')

File ~/frappe-bench/apps/frappe/frappe/init.py:1344, in get_last_doc(doctype, filters, order_by, for_update) 1342 def get_last_doc(doctype, filters=None, order_by="creation desc", *, for_update=False): 1343 """Get last created document of this type.""" -> 1344 d = get_all(doctype, filters=filters, limit_page_length=1, order_by=order_by, pluck="name") 1345 if d: 1346 return get_doc(doctype, d[0], for_update=for_update)

File ~/frappe-bench/apps/frappe/frappe/init.py:2057, in get_all(doctype, *args, *kwargs) 2055 if "limit_page_length" not in kwargs: 2056 kwargs["limit_page_length"] = 0 -> 2057 return get_list(doctype, args, **kwargs)

File ~/frappe-bench/apps/frappe/frappe/init.py:2032, in get_list(doctype, *args, kwargs) 2013 """List database query via frappe.model.db_query. Will also check for permissions. 2014 2015 :param doctype: DocType on which query is to be made. (...) 2028 frappe.get_list("ToDo", fields="", filters = [["modified", ">", "2014-01-01"]]) 2029 """ 2030 import frappe.model.db_query -> 2032 return frappe.model.db_query.DatabaseQuery(doctype).execute(args, kwargs)

File ~/frappe-bench/apps/frappe/frappe/model/db_query.py:185, in DatabaseQuery.execute(self, fields, filters, or_filters, docstatus, group_by, order_by, limit_start, limit_page_length, as_list, with_childnames, debug, ignore_permissions, user, with_comment_count, join, distinct, start, page_length, limit, ignore_ifnull, save_user_settings, save_user_settings_fields, update, add_total_row, user_settings, reference_doctype, run, strict, pluck, ignore_ddl, parent_doctype) 175 kwargs = { 176 "as_list": as_list, 177 "with_comment_count": with_comment_count, (...) 181 "parent_doctype": parent_doctype, 182 } | self.dict 183 return controller.get_list(kwargs) --> 185 self.columns = self.get_table_columns() 187 # no table & ignore_ddl, return 188 if not self.columns:

File ~/frappe-bench/apps/frappe/frappe/model/db_query.py:546, in DatabaseQuery.get_table_columns(self) 544 def get_table_columns(self): 545 try: --> 546 return get_table_columns(self.doctype) 547 except frappe.db.TableMissingError: 548 if self.ignore_ddl:

File ~/frappe-bench/apps/frappe/frappe/model/meta.py:73, in get_table_columns(doctype) 72 def get_table_columns(doctype): ---> 73 return frappe.db.get_table_columns(doctype)

File ~/frappe-bench/apps/frappe/frappe/database/database.py:1197, in Database.get_table_columns(self, doctype) 1195 columns = self.get_db_table_columns("tab" + doctype) 1196 if not columns: -> 1197 raise self.TableMissingError("DocType", doctype) 1198 return columns

ProgrammingError: ('DocType', 'ZATCA Business Settings')

In [2]: settings.is_simulation_server

NameError Traceback (most recent call last) Cell In[2], line 1 ----> 1 settings.is_simulation_server

NameError: name 'settings' is not defined

In [3]:

saud-2 commented 3 months ago

(You can replace get_last_doc with get_doc('ZATCA Business Settings', 'name-of-your-settings') if it's not the last document)

:~/frappe-bench/apps/ksa_compliance/ksa_compliance/ksa_compliance/doctype/zatca_business_settings$ bench console Apps in this namespace: frappe, erpnext

In [1]: settings = frappe.get_doc('ZATCA Business Settings', 'Aziziya branch')

ModuleNotFoundError Traceback (most recent call last) File ~/frappe-bench/apps/frappe/frappe/modules/utils.py:250, in load_doctype_module(doctype, module, prefix, suffix) 249 try: --> 250 doctype_python_modules[key] = frappe.get_module(module_name) 251 except ImportError as e:

File ~/frappe-bench/apps/frappe/frappe/init.py:1476, in get_module(modulename) 1475 """Returns a module object for given Python module name using importlib.import_module.""" -> 1476 return importlib.import_module(modulename)

File /usr/lib/python3.11/importlib/init.py:126, in import_module(name, package) 125 level += 1 --> 126 return _bootstrap._gcd_import(name[level:], package, level)

File :1204, in _gcd_import(name, package, level)

File :1176, in _find_andload(name, import)

File :1126, in _find_and_loadunlocked(name, import)

File :241, in _call_with_frames_removed(f, *args, **kwds)

File :1204, in _gcd_import(name, package, level)

File :1176, in _find_andload(name, import)

File :1140, in _find_and_loadunlocked(name, import)

ModuleNotFoundError: No module named 'frappe.core.doctype.zatca_business_settings'

The above exception was the direct cause of the following exception:

ImportError Traceback (most recent call last) Cell In[1], line 1 ----> 1 settings = frappe.get_doc('ZATCA Business Settings', 'Aziziya branch')

File ~/frappe-bench/apps/frappe/frappe/init.py:1333, in get_doc(*args, kwargs) 1316 """Return a frappe.model.document.Document object of the given type and name. 1317 1318 :param arg1: DocType name as string or* document JSON. (...) 1329 1330 """ 1331 import frappe.model.document -> 1333 doc = frappe.model.document.get_doc(args, **kwargs) 1335 # Replace cache if stale one exists 1336 if not kwargs.get("for_update") and (key := can_cache_doc(args)) and cache.exists(key):

File ~/frappe-bench/apps/frappe/frappe/model/document.py:83, in get_doc(*args, *kwargs) 80 else: 81 raise ValueError('"doctype" is a required key') ---> 83 controller = get_controller(doctype) 84 if controller: 85 return controller(args, **kwargs)

File ~/frappe-bench/apps/frappe/frappe/model/base_document.py:70, in get_controller(doctype) 68 site_controllers = frappe.controllers.setdefault(frappe.local.site, {}) 69 if doctype not in site_controllers: ---> 70 site_controllers[doctype] = import_controller(doctype) 72 return site_controllers[doctype]

File ~/frappe-bench/apps/frappe/frappe/model/base_document.py:95, in import_controller(doctype) 92 module = frappe.get_module(module_path) 94 else: ---> 95 module = load_doctype_module(doctype, modulename) 96 classname = doctype.replace(" ", "").replace("-", "") 98 class = getattr(module, classname, None)

File ~/frappe-bench/apps/frappe/frappe/modules/utils.py:254, in load_doctype_module(doctype, module, prefix, suffix) 252 msg = f"Module import failed for {doctype}, the DocType you're trying to open might be deleted." 253 msg += f"
Error: {e}" --> 254 raise ImportError(msg) from e 256 return doctype_python_modules[key]

ImportError: Module import failed for ZATCA Business Settings, the DocType you're trying to open might be deleted.
Error: No module named 'frappe.core.doctype.zatca_business_settings'

In [2]: settings.is_simulation_server

NameError Traceback (most recent call last) Cell In[2], line 1 ----> 1 settings.is_simulation_server

NameError: name 'settings' is not defined

In [3]:

mhaggag commented 3 months ago

Apps in this namespace: frappe, erpnext

This seems to indicate ksa_compliance is not installed in that site. Do you have multiple sites on the bench?

saud-2 commented 3 months ago

yes I have two sites, I installed it in one site for testing .

mhaggag commented 3 months ago

In that case, you need to run bench --site name-of-your-site console

saud-2 commented 3 months ago

Apps in this namespace: frappe, erpnext, ksa_compliance

In [1]: settings = frappe.get_last_doc('ZATCA Business Settings')

In [2]: settings.is_simulation_server Out[2]: False

In [3]:

saud-2 commented 3 months ago

Apps in this namespace: frappe, erpnext, ksa_compliance

In [1]: settings = frappe.get_doc('ZATCA Business Settings', 'Aziziya branch')

In [2]: settings.is_simulation_server Out[2]: False

In [3]:

mhaggag commented 3 months ago

OK, so it's clearly not detecting it. Either the configured URL is wrong (perhaps it has leading whitespace?) or the URL in the code is wrong. Can you paste settings.fatoora_server_url in the console after the settings = ... line?

saud-2 commented 3 months ago

Apps in this namespace: frappe, erpnext, ksa_compliance

In [1]: settings = frappe.get_last_doc('ZATCA Business Settings')

In [2]: settings.fatoora_server_url Out[2]: ' https://gw-fatoora.zatca.gov.sa/e-invoicing/simulation/'

In [3]:

mhaggag commented 3 months ago

In [2]: settings.fatoora_server_url Out[2]: ' https://gw-fatoora.zatca.gov.sa/e-invoicing/simulation/' Out[2]: ' https://gw-fatoora.zatca.gov.sa/e-invoicing/simulation/'

There's a leading space in the configured URL, before the 'https'. Please remove that, and save the ZATCA Business Settings and try onboarding again.

saud-2 commented 3 months ago

yes you are right . there was a space before URL.

Now I am integrated

Thank you .

mhaggag commented 3 months ago

Thanks for reporting the issue and bearing with us until we resolved it. We'll be fixing the logic to automatically trim the URL before processing it to make sure this trip up someone else.