Docker image?

[semiBE]

Hello, Any plans of containerizing this application yet? I am trying to do it myself but it appears that some dependencies wouldn`t work in production environment (and crash in development as well) Thanks!

[dickwolff]

Thanks for your message! What kind of application/benefit do you see for running the tool inside a docker container versus running it on your local machine?

[semiBE]

Not sure whether it`s a personal preference but I just prefer to keep everything separated while well interconnected. And, ghostfolio has the option to go for a selfhosted docker container and it would pair up pretty well! In any case, thanks a lot guys for your work!

---

Da: Dick Wolff @.> Inviato: mercoledì 31 gennaio 2024 18:38 A: dickwolff/Export-To-Ghostfolio @.> Cc: semiBE @.>; Author @.> Oggetto: Re: [dickwolff/Export-To-Ghostfolio] Docker image? (Issue #25)

Thanks for your message! What kind of application/benefit do you see for running the tool inside a docker container versus running it on your local machine?

— Reply to this email directly, view it on GitHubhttps://github.com/dickwolff/Export-To-Ghostfolio/issues/25#issuecomment-1919590866, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB3GOCJAJSGOVLHEJEO54DTYRJ6PRAVCNFSM6AAAAABCTO6BIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGU4TAOBWGY. You are receiving this because you authored the thread.

[dickwolff]

I could do a poc with a file watcher that maps to a local folder on your disk where you could put your export and then the tool would import it, process and then output somewhere.

I will try to put something together when I have some time and will tag you whenever so you can test it!

[dickwolff]

I have created a first version of the tool that should support DEGIRO and Trading 212 conversions.

You can run it like docker run -d -v /C/users/dickw/docker_in:/var/input -v /C/users/dickw/docker_out:/var/output dickwolff/export-to-ghostfolio

The containerized version of the tool will look to /var/input within the container (which you can specify with a volume mount) for input files, and likewise to var/output for the folder where the result file will be put.

The Docker image will continue to look for new files and when a file is given it will try to detect which converter is needed to parse and then run the same script as when you run npm run start xxx locally.

Can you check wether this is what you were expecting @semiBE , that way I know I'm on the right path and can then continue on improving the solution afterwards.

Thanks!

Please note this is a first draft and there is no additional error handling and such 😅

Edit: I forgot about the Ghostfolio account id and haven't tested this myself if it works with a Docker environment variable.

[semiBE]

Hey @dickwolff thanks for reacting so quickly! The PoC I think is pretty good I think! In case you don't want to have it running all the time you can just spin it to process the available files and stop it afterwards! I have spun it up once and tried to copy one degiro csv file in the input folder but it looks like there's nothing appearing in the log somehow. I have integrated the env variables in my docker composer (not the INPUT_FILE of course) that you can see here below

version: '3.3'
services:
  ExportToGhostfolio:
    image: dickwolff/export-to-ghostfolio
    container_name: ExportToGhostfolio
    volumes:
      - /srv/dev-...../ExportToGhostfolio/input:/var/input
      - /srv/dev-...../ExportToGhostfolio/output:/var/output
    restart: unless-stopped

[dickwolff]

Maybe making it watch files should be a flag you give when starting the container. Will look into that when I have time.

Will then also look at the issue you mention. I did some code merging today (merged a unit test pr early because i had some code changes in there for better error handling which could benefit this branch). I haven't run the container since, so maybe an error slipped through.

[dickwolff]

@semiBE I seem to have changed the folder names from "/var/input" to "/var/e2g-input" and likewise for output. It makes sense your files aren't processed then ;-)

Try again with the like of: docker run -d -v /C/users/dickw/docker_in:/var/e2g-input -v /C/users/dickw/docker_out:/var/e2g-output dickwolff/export-to-ghostfolio

[dickwolff]

Accidental closing by mentioning in PR #27

I have merged the first version of the tool. Any additional feedback would be welcome and I can add that in another PR.

[semiBE]

Hi! So I have tested this new version and indeed that it recognizes a new file which nevertheless fails the conversion (log below). Note that my degiro csv file is in French if that makes any difference and I don't see a field called "comment" either. The csv is directly downloaded from the website and passed on to the e2g.

2024-02-02T18:42:17.682298425Z [i] Found Account.csv!
2024-02-02T18:42:17.686476537Z [i] Determined the file type to be of kind 'degiro'.
2024-02-02T18:42:17.686651631Z [i] Processing file using DeGiro converter
2024-02-02T18:42:17.686663435Z [i] NOTE: There is a new version available of the DeGiro converter
2024-02-02T18:42:17.686683270Z [i] The new converter has multiple record parsing improvements and also supports platform fees.
2024-02-02T18:42:17.686690394Z [i] The new converter is currently in beta and we're looking for your feedback!
2024-02-02T18:42:17.686700312Z [i] You can run the beta converter with the command 'npm run start degiro-v2'.
2024-02-02T18:42:17.700139943Z [i] Read CSV file. Start processing..
2024-02-02T18:42:20.150193827Z /app/src/converters/degiroConverter.ts:156
2024-02-02T18:42:20.150217504Z             if (result.activities[result.activities.length - 1].comment === "txfees") {
2024-02-02T18:42:20.150222672Z                                                                 ^
2024-02-02T18:42:20.151421098Z TypeError: Cannot read properties of undefined (reading 'comment')
2024-02-02T18:42:20.151449943Z     at DeGiroConverter.<anonymous> (/app/src/converters/degiroConverter.ts:156:65)
2024-02-02T18:42:20.151457067Z     at Generator.next (<anonymous>)
2024-02-02T18:42:20.151463213Z     at fulfilled (/app/src/converters/degiroConverter.ts:5:58)
2024-02-02T18:42:20.151467473Z     at processTicksAndRejections (node:internal/process/task_queues:95:5)

[dickwolff]

There is a chance there is some french language specific stuff in your export. Can you maybe send me a (partial) export example so I can take a look?

[semiBE]

Indeed it could be. There you go:

Date,Heure,Date de,Produit,Code ISIN,Description,FX,Mouvements,,Solde,,ID Ordre
03-01-2024,19:47,03-01-2024,,,"Virement depuis votre Compte Espèces à la flatexDEGIRO Bank: 1 0,84 EUR",,,,EUR,0.21,
03-01-2024,19:47,03-01-2024,FLATEX EURO BANKACCOUNT,NLFLATEXACNT,Degiro Cash Sweep Transfer,,EUR,1.84,EUR,1.05,
03-01-2024,14:26,03-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,Taxe sur les Opérations Boursières belge (TOB),,EUR,-0.10,EUR,6.21,1dgsag-ddgsc-4dsgd-9mh6-5aaaaaaa

[dickwolff]

Thanks. I will add those to the DEGIRO converters.

Do you perhaps also have a buy/sell (+tx fee) and dividend (+tax) records? So I can check whether those will work or if they also require a code change.

[semiBE]

You can try to add these as well. I am not sure whether I have any position with dividends actually..

12-01-2024,11:10,12-01-2024,,,Dépôt flatex,,EUR,20.00,EUR,74.21,
03-01-2024,14:26,03-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,Frais DEGIRO de courtage et/ou de parties tierces,,EUR,-1.00,EUR,6.51,1e2a-d-4-9d5b834

[rodriguestiago0]

Hey, I have the same issue as mention aboved. Mine is portuguese.

Data,Hora,Data Valor,Produto,ISIN,Descrição,T.,Mudança,,Saldo,,ID da Ordem
05-02-2024,22:52,05-02-2024,,,"Levantamentos da sua Conta Caixa na flatexDEGIRO Bank AG: 2,5 EUR",,,,EUR,15.68,
05-02-2024,22:52,05-02-2024,FLATEX EURO BANKACCOUNT,NLFLATEXACNT,Degiro Cash Sweep Transfer,,EUR,2.50,EUR,18.18,
05-02-2024,07:17,31-01-2024,,,Custo de Conectividade DEGIRO 2024 (Euronext Amsterdam - EAM),,EUR,-2.50,EUR,15.68,

I cloned the repo and tried it locally after trying the docker version and both failed.

Also on docker version I can force to use degiro-v2.

Edit: With a small file version 1 works with docker.

This line is breaking the parser:

15-05-2019,09:05,15-05-2019,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 6 ISHARES MSCI WOR A@49,785 EUR (IE00B4L5Y983)",,EUR,-298.71,EUR,0.64,a47e2746-bfbd-4654-bd6c-5e58e470d32f

Degiro-v2 is throwing this error

Error: ENOENT: no such file or directory, open 'C:\Users\Rodrigues\Documents\Export-To-Ghostfolio\Data,Hora,Data Valor,Produto,ISIN,Descrição,T.,Mudança,,Saldo,,ID da Ordem
15-05-2019,09:05,15-05-2019,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 6 ISHARES MSCI WOR A@49,785 EUR (IE00B4L5Y983)",,EUR,-298.71,EUR,0.64,a47e2746-bfbd-4654-bd6c-5e58e470d32f'```

[dickwolff]

@rodriguestiago0 Thanks for providing the lines; I will add those to the converter whenever I have some time and provide an update here then!

[rodriguestiago0]

Something strage. Maybe I'm not aware on the parser works.

Data,Hora,Data Valor,Produto,ISIN,Descrição,T.,Mudança,,Saldo,,ID da Ordem
02-01-2024,14:42,02-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,Comissões de transação DEGIRO e/ou taxas de terceiros,,EUR,-1.00,EUR,2.54,7b377a93-5695-4131-8954-5c78996fbed4
02-01-2024,14:42,02-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 1 ISHARES MSCI WOR A@82,055 EUR (IE00B4L5Y983)",,EUR,-82.06,EUR,3.54,7b377a93-5695-4131-8954-5c78996fbed4

The example above works. But the example below doesn't.

 Data,Hora,Data Valor,Produto,ISIN,Descrição,T.,Mudança,,Saldo,,ID da Ordem
02-01-2024,14:42,02-01-2024,ISHARES MSCI WOR A,IE00B4L5Y983,"Compra 1 ISHARES MSCI WOR A@82,055 EUR (IE00B4L5Y983)",,EUR,-82.06,EUR,3.54,7b377a93-5695-4131-8954-5c78996fbed4

[dickwolff]

@rodriguestiago0

Degiro-v2 is throwing this error

I made some changes earlier that puts the contents of the CSV as the parameter of the process function of the converter, instead of the file path. I made those changes everywhere but the DEGIRO V2 converter. I have fixed this and the tool now runs.

The example above works. But the example below doesn't.

I found an issue with the look-ahead mechanism that sometimes had empty records. That made the tool crash (without an obvious error). I added a check to prevent this. The tool now produces a valid result.

[dickwolff]

I have also added a new environment variable FORCE_DEGIRO_V2 which you can set to true. When you provide a DEGIRO file, the converter will use the V2 converter instead of the normal one.

[dickwolff]

@semiBE I have added French language specifics to DEGIRO V2 converter, which is now also published via Docker (as are the Portugese ones @rodriguestiago0). Can you try if the converter now works for you?

Edit 21:23 > I had an error in my workflow file that caused the workflow to fail, so there was no new image pushed. I have fixed the workflow and there now is a new image available!

[rodriguestiago0]

@dickwolff thank you.

Tried with v2 and it's no writing the output. And v1 fails with the same error

[dickwolff]

I'm sorry to hear that. I will look into this weekend and get back to you.

[semiBE]

IT WORKS! The problem in my case was that some lines in the csv were not correctly parsed and it was giving some error at specific but those lines didn`t correspond to operations but rather to some cash management so I could just delete them. Now it parses them perfectly :) All the operations are in ghostfolio! Love it! PS: I might spend some time in the next couple of months to develop a code to import etoro operations but I will only be able to do to in python, would you be interested in including it into this tool? Thanks a lot @dickwolff!

[dickwolff]

Great to hear it works now! I can also add those cash management records to the ignore list and then you dont have to remove them yourself next time. But that is up to you.

Regarding eToro, if you can provide me with an example csv with all transactions types it has (buy, sell, dividend, fees if applicable), I think I can add a converter in a couple of hours 😅

[semiBE]

Well, these lines gave issues:

05-02-2024,07:17,31-01-2024,,,Frais de connexion aux places boursières 2024 (Euronext Amsterdam - EAM),,EUR,-2.50,EUR,1211.21,
05-02-2024,07:17,31-01-2024,,,Frais de connexion aux places boursières 2024 (Xetra - XET),,EUR,-2.50,EUR,1213.71,
30-06-2023,08:00,29-06-2023,,,Operation de change - Crédit,,EUR,21.70,EUR,6.20,
15-02-2023,16:07,15-02-2023,,,Versement de fonds,,EUR,0.05,EUR,1.50,

For eToro there`s no csv only xlsx because it has multiple tabs but only one is the interesting one. Here the content of a converted xlsx to csv:

Date,Type,Details,Amount,Units,Realized Equity Change,Realized Equity,Balance,Position ID,Asset type,NWA
01/01/2024 05:50:54,Interest Payment,,0.08,-,0.08,"4,581.91",0.00,-,,0.00
02/01/2024 00:10:33,Dividend,NKE/USD,0.17,-,0.17,"4,581.91",99.60,2272508626,Stocks,0.00
02/01/2024 00:10:33,Dividend,NKE/USD,0.16,-,0.16,"4,581.91",99.60,2446419144,Stocks,0.00
03/01/2024 00:04:03,Dividend,LRCX/USD,0.51,-,0.51,"4,581.91",100.11,1434892477,Stocks,0.00
03/01/2024 13:09:09,Withdraw Fee,-,0.00,-,0.00,"4,581.91",100.11,-,,0.00
03/01/2024 13:09:09,Withdrawal Conversion Fee,-,-1.36,-,0.00,"4,581.91",0.00,-,,0.00
03/01/2024 13:09:09,Withdraw Request,-,-100.11,-,-100.11,"4,581.91",0.00,-,,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.28,-,0.28,"4,581.91",0.00,1446737312,Stocks,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.04,-,0.04,"4,581.91",0.00,1453559680,Stocks,0.00
09/01/2024 15:30:40,Position closed,OLED/USD,18.43,0.102626,7.37,"4,581.91",0.00,2355395242,Stocks,0.00
09/01/2024 15:36:54,Position closed,AMAT/USD,0.38,0.002556,0.04,"4,581.91",0.00,2596568830,Stocks,0.00
09/01/2024 15:37:16,Open Position,AMD/USD,49.88,0.337209,0.00,"4,581.91",0.00,2596572937,Stocks,0.00
09/01/2024 15:37:25,Open Position,SLAB/USD,27.37,0.219575,0.00,"4,581.91",0.00,2596576844,Stocks,0.00
09/01/2024 15:40:09,Open Position,SWKS/USD,93.62,0.888488,0.00,"4,581.91",0.00,2596634782,Stocks,0.00
10/01/2024 00:11:24,Dividend,DIS/USD,0.14,-,0.14,"4,581.91",0.00,1003296596,Stocks,0.00
10/01/2024 00:11:24,Dividend,DIS/USD,0.29,-,0.29,"4,581.91",0.00,1068527084,Stocks,0.00
16/01/2024 00:17:08,Dividend,HST/USD,0.07,-,0.07,"4,581.91",0.80,2565075017,Stocks,0.00
16/01/2024 20:08:09,Position closed,AMD/USD,61.27,0.388423,38.57,"4,581.91",0.80,2602058508,Stocks,0.00
17/01/2024 00:04:27,Dividend,KER/EUR,0.03,-,0.03,"4,581.91",0.80,697085768,Stocks,0.00
17/01/2024 00:04:28,Dividend,KER/EUR,0.01,-,0.01,"4,581.91",0.80,831716072,Stocks,0.00
17/01/2024 00:04:28,Dividend,KER/EUR,0.01,-,0.01,"4,581.91",0.80,1074146904,Stocks,0.00

Anything which can come out of this will be great, thanks in advance!

[dickwolff]

Quick question, I don't see a currency in the CSV. What is de default currency that Etoro uses? And are investments converted to that currency? (For example there is a KER/EUR and DIS/USD; different currencies).

Well, these lines gave issues:

These should now be processed by the tool, so no need to remove them in the future!

[semiBE]

The default currency is USD and for each transaction you have that it's processed in its own currency and you guessed correctly it's after that backslash. That was fast! I'll provide you with the instructions on how to export the account statements from eToro so that you can add them to the Readme. Thanks again!

[semiBE]

I can add that also this line causes issues:

30-06-2023,08:00,29-06-2023,,,Operation de change - débit,,EUR,21.70,EUR,6.20,

[dickwolff]

I have added débit to the skip list.

[dickwolff]

Tried with v2 and it's no writing the output. And v1 fails with the same error

@rodriguestiago0 I have run the tool locally with the file you provided and added some more language specific rules to the DEGIRO v2 converter. The tool now runs succesfully. It is also available as part of the 0.4.0-beta container on Docker hub.

[rodriguestiago0]

Thank you for fix

On Fri, 9 Feb 2024 at 08:48, Dick Wolff @.***> wrote:

Tried with v2 and it's no writing the output. And v1 fails with the same error

@rodriguestiago0 https://github.com/rodriguestiago0 I have run the tool locally with the file you provided and added some more language specific rules to the DEGIRO v2 converter. The tool now runs succesfully. It is also available as part of the 0.4.0-beta container on Docker hub.

— Reply to this email directly, view it on GitHub https://github.com/dickwolff/Export-To-Ghostfolio/issues/25#issuecomment-1935538238, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVNAC6Q2K4UBBE7GTYF4KTYSXPF7AVCNFSM6AAAAABCTO6BIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZVGUZTQMRTHA . You are receiving this because you were mentioned.Message ID: @.***>

[dickwolff]

Closing this with fixes from #28 .

Should there be any other issue, please open a new issue.