search

dickwolff / Export-To-Ghostfolio

Convert transaction history export from your favorite broker to a format that can be imported in Ghostfolio.
https://hub.docker.com/r/dickwolff/export-to-ghostfolio
Apache License 2.0
45 stars 8 forks source link

eToro converter - transactions with same `Details` not counted? #29

Closed semiBE closed 7 months ago

semiBE commented 7 months ago

Hi! I am trying to convert my January transactions and there are multiple transactions (with the same name) which are happening. It looks like the converter only took in consideration the first of the list and then it jumps to the next one which has a different value in the Details field.

In the example below it will only count the first NXPI/USD dividend and the first OLED/USD position closed.

Date,Type,Details,Amount,Units,Realized Equity Change,Realized Equity,Balance,Position ID,Asset type,NWA
07/01/2024 13:21:57,Dividend,NXPI/USD,0.02,-,0.02,"45,3.00",0.00,1520124620,Stocks,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.02,-,0.02,"45,3.02",0.00,1539539526,Stocks,0.00
07/01/2024 13:21:57,Dividend,NXPI/USD,0.03,-,0.03,"45,3.05",0.00,2472065850,Stocks,0.00
09/01/2024 15:30:40,Position closed,OLED/USD,18.43,0.102626,7.37,"45,0.42",0.00,2355395242,Stocks,0.00
09/01/2024 15:30:40,Position closed,OLED/USD,8.87,0.049420,1.80,"45,2.22",0.00,2472033271,Stocks,0.00
09/01/2024 15:30:41,Position closed,OLED/USD,89.92,0.500702,4.31,"45,6.53",0.00,1446690287,Stocks,0.00

Thanks! for checking this out!

dickwolff commented 7 months ago

When I take the main source code and run npm run start etoro with the input you provided above, I get the following result:

{
    "meta": {
        "date": "2024-02-09T11:58:39.900Z",
        "version": "v0"
    },
    "activities": [
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 1,
            "type": "DIVIDEND",
            "unitPrice": 0.02,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-07T13:21:57+01:00",
            "symbol": "NXPI"
        },
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 1,
            "type": "DIVIDEND",
            "unitPrice": 0.02,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-07T13:21:57+01:00",
            "symbol": "NXPI"
        },
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 1,
            "type": "DIVIDEND",
            "unitPrice": 0.03,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-07T13:21:57+01:00",
            "symbol": "NXPI"
        },
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 0.102626,
            "type": "SELL",
            "unitPrice": 18.43,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-09T15:30:40+01:00",
            "symbol": "OLED"
        },
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 0.04942,
            "type": "SELL",
            "unitPrice": 8.87,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-09T15:30:40+01:00",
            "symbol": "OLED"
        },
        {
            "accountId": "c2a3093b-beef-4232-9be3-5b28bc3dcdad",
            "comment": "",
            "fee": 0,
            "quantity": 0.500702,
            "type": "SELL",
            "unitPrice": 89.92,
            "currency": "USD",
            "dataSource": "YAHOO",
            "date": "2024-01-09T15:30:41+01:00",
            "symbol": "OLED"
        }
    ]
}

Which configuration did you try to run the tool with?

semiBE commented 7 months ago

Interesting.. I was using the 0.4.1-beta. I have then reverted back to 0.4.0 and 0.4.0-beta but it appears there are too many queries to Yahoo and it timed out.. will have to try later on.

2024-02-09T12:57:24.558282395Z [i] Determined the file type to be of kind 'etoro'.
2024-02-09T12:57:24.564313286Z [i] Processing file using Etoro converter
2024-02-09T12:57:24.593645697Z [i] Read CSV file. Start processing..
2024-02-09T12:57:25.005825926Z 
2024-02-09T12:57:25.005855260Z [e] An error ocurred while trying to retrieve symbol NKE/USD (line 5)!
2024-02-09T12:57:25.005860917Z 
2024-02-09T12:57:25.005935649Z [e] An error ocurred while processing.
2024-02-09T12:57:25.005963586Z [e] Error details: FetchError: invalid json response body at https://query2.finance.yahoo.com/v1/finance/search?lang=en-US&region=US&quotesCount=10&newsCount=0&enableFuzzyQuery=false&quotesQueryId=tss_match_phrase_query&multiQuoteQueryId=multi_quote_single_token_query&newsQueryId=news_cie_vespa&enableCb=true&enableNavLinks=true&enableEnhancedTrivialQuery=true&q=null reason: Unexpected token 'T', "Too Many Requests
dickwolff commented 7 months ago

Sometimes Yahoo Finance times you out for a bit. Usually within 15 minutes you will be able to try again.

I have been working on a caching mechanism for a bit (see #22 ) that caches symbols so you dont have to ping Yahoo Finance every time.

semiBE commented 7 months ago

Sometimes Yahoo Finance times you out for a bit. Usually within 15 minutes you will be able to try again.

It looks like another error pops up: 

2024-02-09T14:31:03.583308695Z [e] Error details: FetchError: invalid json response body at https://query2.finance.yahoo.com/v1/finance/search?lang=en-US&region=US&quotesCount=10&newsCount=0&enableFuzzyQuery=false&quotesQueryId=tss_match_phrase_query&multiQuoteQueryId=multi_quote_single_token_query&newsQueryId=news_cie_vespa&enableCb=true&enableNavLinks=true&enableEnhancedTrivialQuery=true&q=IBE reason: Unexpected token '<', "<!doctype "... is not valid JSON

I have been working on a caching mechanism for a bit (see #22 ) that caches symbols so you dont have to ping Yahoo Finance every time.

Is it in the 0.5.0-beta docker img by chance?

Thanks!

dickwolff commented 7 months ago

It looks like another error pops up

The error doesn’t ring a bell. The only thing that comes to mind is yahoo finance not being available (since the response looks to be html). Can you maybe try again? Will think about some error handling regarding Yahoo finance in the future..

Is it in the 0.5.0-beta docker img by chance?

It is 🙂

semiBE commented 7 months ago

Is it in the 0.5.0-beta docker img by chance?

It is 🙂

0.5.0-beta works like a charm!

I noticed one error in the conversion though. The column Amount in the eToro csv is not the Unit Price. The Unit Price would be Amount\Units.

dickwolff commented 7 months ago

Great to hear!!

The Unit Price would be Amount\Units.

Gotcha, will fix that as part of 0.5.0!

dickwolff commented 7 months ago

I have just pushed the fix for unit price calculations! When you pull the latest 0.5.0-beta image you'll be able to check it out!