N8N: Integration Development

paulwer commented 1 year ago

I am currently pushing changes to the automation-platform n8n for an updated version of the existing node. The current node supports v4 and v5 of invoiceninja apis. My extension is focused on the features of v5, explicitly on the new webhooks and fields.

N8N is like zapier, but with the benefit, that it is open-source and can be self-hosted.

Is it possible / desired to review the changes with feedback. Is it an option to integrate N8N like Zapier in the future. Do to the open-source aspect of invoiceninja, I would find it highly suiteable.

turbo124 commented 1 year ago

@paulwer we can certainly assist with any questions.

paulwer commented 1 year ago

@turbo124 I cannot check if the node still works for v4, so therefore it would be great, when someone can check this afterwards to make it compatible or do some testing with my current state of the implementation.

Feedback for more features to add are also welcome.

A check of the interfaces would also be great. (Are the spellings of the parameters right / are the ids of the events right / am I missing something [like events or parameters])

turbo124 commented 1 year ago

@paulwer

is it possible to version the changes specifically for v5?

paulwer commented 1 year ago

@turbo124

currently the code base combines v4 and v5 props, but I will take the oportunity to split the codebased more strictly into v4 and v5. I will come back here, as these changes are complete.

Can you provide TypeScript interfaces for me for v5 (and v4 if you want) in the meantime?

turbo124 commented 1 year ago

https://github.com/invoiceninja/ui/tree/main/src/common/interfaces

These are for v5 we don't have v4

paulwer commented 1 year ago

do you also have DTOs for the api-calls?

turbo124 commented 1 year ago

@paulwer

Unfortunately nothing documented as yet.

paulwer commented 1 year ago

therefore, i will use the entities. for each method I will not support all fields.

Another question: when I am creating a new object invoice/client etc. why is there an existing endpoint for getting a template first? Is it necessary to merge the template with the data, i would like to pass to the api?

turbo124 commented 1 year ago

the create route is just a convenience route, our React app uses this to stub new entities

paulwer commented 1 year ago

thanks. After a while of local development, i guess a hard split of the nodes to manage them individualy is better suiteable. I am waiting for a response of n8n to know the best next steps to be taken. I will come back here, if there are new informations on this topic or any help needed.

Is it possible, to refer n8n in the future as an alternative to zapier? Exspecialy for the self-hosting folks this would be great news. Do you need anything here to prepare?

turbo124 commented 1 year ago

@paulwer

We are all about free alternatives, so we'd definitely include this in the readme.

paulwer commented 1 year ago

@turbo124 Is there any kind of pagination implemented in the v5 apis? I did not find anything within swagger. In swagger there are also no DTOs specified.

paulwer commented 1 year ago

https://github.com/invoiceninja/ui/tree/main/src/common/interfaces

These are for v5 we don't have v4

Please make sore to notify me, for any breaking changes of the interfaces in the future, so I can do the necessary changes for the node afterwards.

hillelcoren commented 1 year ago

You can find info about query params here:

https://invoiceninja.github.io/docs/api/clients/#query-parameters

paulwer commented 1 year ago

are these query params the same for all resources?

turbo124 commented 1 year ago

@paulwer

For pagination these are available across all resources

per_page: The number of clients per page you want returned page: The page number

includes are also available across all resources include: A comma separated list of relations to include ie. contacts,documents,gateway_tokens

For each entity there are a range of available filters depending on the entity type.

you can inspect these in app/Filters

paulwer commented 1 year ago

Is there a default page size? Or parameter behavior to disable pagination?

turbo124 commented 1 year ago

20 is the default

paulwer commented 1 year ago

@turbo124

I extended the resources availabe to the following:

Client
Expense
Invoice
Payment
Project
Quote
Task
Vendor
Product
Recurring Invoices
Recurring Expenses
Recurring Quotes

I am planing to review the Abonements endpoints also, to see if there are any important resources to implement. Did I miss something important?

As mentioned, this only affects V5 support: V4 I will stay as exists. (Client, Invoice, Expense, Payment. Quote, Task)

turbo124 commented 1 year ago

@paulwer

A few missing from the list includes::

Credits Transactions Subscriptions Purchase Orders Documents

paulwer commented 1 year ago

@turbo124 Is there an entity for recurring quotes?

paulwer commented 1 year ago

@hillelcoren @turbo124 please take a look at my fork and do the following checks/tasks:

In N8N we have to define the mask of parameters for the node. Therefore these files are used to display the ui. Currently I've created these files based on my experience with the api and the provided informations (swagger, interfaces, look at filters). The files are structured based on resources and operations (both dropdowns in the ui) For each resource, I support up to 7 operations:

get
getAll
create
update
delete
download - getting the document of an invoice (etc)
actions - do actions, like in api (ex. mail, clone, archive, restore etc)

**1. Please verify for each ressource/operation, that all provided parameters in editable sections (create, update, delete) are all editable through the INv5-API (DTO) and not only a data-representation field, which get processed and shown in the ui

If you are able to, provide me with details about required parameters for create/update operations for each ressource
Start testing the node by running the branch localy
Feel free to provide feedback**

Workflow (review files)

go into V3-directory (please ignore directory V1u2, thats for N8N old node versions, to prevent breaking changes in existing projects)
go into v5-directory (thats the InvoiceNinja V5 part of the node)
go into the files Description.ts
Do checks like described above
Do a file change request / comment, if you have feedback Link: https://github.com/paulwer/n8n/tree/feature-invoice-ninja/packages/nodes-base/nodes/InvoiceNinja

Workflow (Testing)

checkout fork paulwer/n8n
run commands:
- initial start of project: pnpm install && pnpm build
- start server: pnpm dev
- I encountered the problem, that the editor doesnt connect to the api, then you have to restart the service

N8N Documentation

Here you find a documentation, how the description-fields are structured. https://docs.n8n.io/integrations/creating-nodes/build/reference/ui-elements/

In Most Cases there are fields under a collection called Additional parameters. (these should be the optional parameters) the data is grouped by the operation and easily findable by searching : (its a comment)

Current Status

-> implemented and first check of parameters done by myself (if it's marked, you can start reviewing it)

[X] Client
[X] Expense
[X] Invoice
[X] Payment
[X] Project
[X] Quote
[X] Task
[X] Vendor
[X] Product
[X] Recurring Invoices
[X] Recurring Expenses
[ ] Recurring Quotes - pending for answer about entities
[x] Credits
[x] Transactions
[x] Subscriptions
[x] Purchase Orders
[x] Documents - pending response

Feel free to document your finished resources in a comment below, so others can pick, which they want to review

Other TODOs

[ ] Check docs (public, check introduction-text, not the ressources, thats my task) https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.invoiceninja/
[ ] Check note within v4: I had added a note saying: Please confirm, if thats ok / wanted. You are using V4 of InvoiceNinja Considder migrating to V5 to have even more resources and operations supported for this node. <a href="https://invoiceninja.com/migrate-to-invoice-ninja-v5/">https://invoiceninja.com/migrate-to-invoice-ninja-v5/</a>

paulwer commented 1 year ago

@paulwer

A few missing from the list includes::

Credits Transactions Subscriptions Purchase Orders Documents

@turbo124 what do you mean by using documents? this is a part of each resource and not a standalone thing? I currently implement operations of all resources, which support downloading an document and output it as an binary attachment within n8n to work with it in the next steps of the flow. Docs Preview Resources and Operations Docs Preview Events (Webhooks)

Do you mean uploading files?

paulwer commented 1 year ago

@hillelcoren @turbo124 the main work is done and all resources above (see list) are created. I will now start testing for my self and waiting for response/feedback, as described above.

Thats a big task to do (I know), so you may provide me a time frame, in which you can start reviewing?

Feel free to ask questions or invite others to contribute. Thanks.

turbo124 commented 1 year ago

@paulwer we wouldn't have the bandwidth to support testing the integration unfortunately. We can assist with technical direction and any specific questions you have.

paulwer commented 1 year ago

@turbo124 there are still open questions (recurring quotes / documents)

What is the difference between client.name and client.display_name (As of my test, I cannot change display_name, because it is processed of name?)

turbo124 commented 1 year ago

@paulwer

display_name is readonly, we don't require a client name, however we always need to be able to display a client name, so we calculate the display_name as we know it will always return a string, the order of returned results is:

client.name or contact.name or contact.email

We have not yet implemented Recurring Quotes, these are only stubbed for now.

In regards to documents, you are able to attach a document to any PUT request. ie /api/v1/invoices this will allow uploading of documents/files to any of the entities.

Documents are also a standalone resource mainly for read/delete operations.

paulwer commented 1 year ago

@turbo124 is it possible to perform an operation, which requires a password like purging clients or deleting documents with an api-token?

I get an error here, that the password is missing.

@turbo124 the delete document ressource is missing in the swagger docs

paulwer commented 1 year ago

@turbo124 the download function also gets a timeout each time, i try to call it through the api. The frontend downloads it through https:/subdomain.domain.de/documents/{hash}.pdf

but not from the api. fix necessary / possible? the link to call is provided within the json-body of the document, so it should work. or be replaced.

turbo124 commented 1 year ago

@paulwer

There are some special routes that required password confirmations using teh X-API-PASSWORD header.

These would be for destructive actions such as purging a company or client also adding/removing a user.

In regards to the downloads, you may want to inspect the response, it is probably streaming the download?

paulwer commented 1 year ago

@turbo124 please check the endpoint yourself. I even get a timeout in postman. What Iam doing wrong here?

turbo124 commented 1 year ago

just tested here myself and it works fine, i just send a GET request with no additional headers.

paulwer commented 1 year ago

@turbo124 server logs:

2023/02/07 07:30:30 [error] 30#30: *653 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.20.0.2, server: _, request: "GET /api/v1/documents/Wpmbk5ezJn/download HTTP/1.1", upstream: "fastcgi://172.20.0.4:9000", host: "subdomain.domain.de"

[2023-02-07 07:48:54] production.ERROR: file_get_contents(https://subdomain.domain.de/storage/IRt64zbj4Q3XAJIFWF933EgTEHM43qcy/documents/Lam8yz9RgFwhhlopX58VBmjJC997rL3U73ty7hTt.pdf): Failed to open stream: Operation timed out {"userId":1,"exception":"[object] (ErrorException(code: 0): file_get_contents(https://subdomain.domain.de/storage/IRt64zbj4Q3XAJIFWF933EgTEHM43qcy/documents/Lam8yz9RgFwhhlopX58VBmjJC997rL3U73ty7h$
[stacktrace]
#0 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(266): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}()
#2 /var/www/app/app/Http/Controllers/DocumentController.php(125): file_get_contents()
#3 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/ResponseFactory.php(145): App\\Http\\Controllers\\DocumentController->App\\Http\\Controllers\\{closure}()
#4 /var/www/app/vendor/symfony/http-foundation/StreamedResponse.php(92): Illuminate\\Routing\\ResponseFactory->Illuminate\\Routing\\{closure}()
#5 /var/www/app/vendor/symfony/http-foundation/Response.php(377): Symfony\\Component\\HttpFoundation\\StreamedResponse->sendContent()
#6 /var/www/app/public/index.php(61): Symfony\\Component\\HttpFoundation\\Response->send()
#7 {main}
"}

Can you may help here, what to do?

turbo124 commented 1 year ago

no sure, try using CURL from the command line.

you'll want to redact your URL from the error message.

the error there is your system is unable to pull in the required file.

paulwer commented 1 year ago

@turbo124 how do I fix this or avoid it in the first place? its a docker installation on arm. oracle linux 9

paulwer commented 1 year ago

@turbo124 I was not able to fix this, therefore i implemented it with fetching from https://subdomain.domain.de/documents/{hash}

This works fine at my end. Is there something to considder with this approach?

paulwer commented 1 year ago

@turbo124 please check the last part of this message Section: TODO

if this is all suiteable or if something should be changed/added/removed

turbo124 commented 1 year ago

@paulwer

Thanks it all looks ok, however please a small change.

current text:

Invoice Ninja is a free open-source online invoicing app for freelancers & businesses. It offers invoicing, payments, expense tracking, & time-tasks.

preferred text:

Invoice Ninja is a free "source available" online invoicing app for freelancers & businesses. It offers invoicing, payments, expense tracking, & time-tasks.

o-psi commented 1 year ago

@paulwer We use N8N for automation and Invoice Ninja for billing. If there is anything we can do to help, including dev servers, please reach out. We have extra capacity in our datacenter.

paulwer commented 1 year ago

@turbo124 @hillelcoren my main work + testing is done.

Please read the updated docs again and check if the main ressources are implemented and if something important is missing or could/should be added. Docs Preview Resources and Operations Docs Preview Events (Webhooks)

hillelcoren commented 1 year ago

@paulwer This looks amazing, thank you for working on it!

turbo124 commented 1 year ago

@paulwer

I think it looks good, just a couple of fixes.

Instead of Invoice Delayed, i think you may mean the Webhook::EVENT_LATE_INVOICE which means the invoice is overdue

In the actions docs, when you refer to

Perform actions on an client (ex. merge, archive, restore, purge, client_statement)

does ex mean exclude? I would have thought you could certainly do all of these except merge/purge

paulwer commented 1 year ago

@turbo124 ex. meant example. I fixed it to i.e. (thanks)

i fixed the spelling for the late event, this woulnd had effected the functionality, but now its a bit more clearer for the viewer.

I also added a property password for the credentials section, so purge client and delete document are also possible.

paulwer commented 1 year ago

@hillelcoren are there plans to extend functionality for zapier or make?

both of them dont support all operations and events.

hillelcoren commented 1 year ago

Once the new events are deployed to production we'll work on a new Zapier release.

Make handle the integration themselves.

paulwer commented 1 year ago

@hillelcoren @turbo124 the docker image is still not updated. is this something you can do?

turbo124 commented 1 year ago

@paulwer

I am working on this now, something is blocking me from tagging, as soon as it is ready I'll tag.

paulwer commented 1 year ago

@turbo124

Status-Update: investigation from the n8n-team of the node is still pending

fyi: description of node was changed: https://github.com/n8n-io/n8n-docs/pull/1033/commits/9af5bdad63c1fa9efa06a0e3f9a84b077d4d5715

The Invoice Ninja node allows you to automate work in Invoice Ninja, and integrate Invoice Ninja with other applications. n8n has built-in support for a wide range of Invoice Ninja features, including creating, updating, deleting, and getting clients, expense, invoice, payments and quotes. 

On this page, you'll find a list of operations the Invoice Ninja node supports and links to more resources.

turbo124 commented 10 months ago

@paulwer Can we close this now?

paulwer commented 10 months ago

I hope not, I already completed the integration completly like half a year ago, but the dev of n8n is based on his responses still working on some changes.

Tbh, I hope they find some time to complete it soon, as I also want to conclude this. Maybe some expression of interest from IN could create more momentum, but I guess they are heavily overloaded currently.

invoiceninja / invoiceninja