search

dickwolff / Export-To-Ghostfolio

Convert transaction history export from your favorite broker to a format that can be imported in Ghostfolio.
https://hub.docker.com/r/dickwolff/export-to-ghostfolio
Apache License 2.0
31 stars 8 forks source link

Swchab conversion error #75

Closed adrianbn closed 1 month ago

adrianbn commented 1 month ago

Tried with multiple accounts and I always get the following output:

docker run --rm -v $PWD/exports/:/var/tmp/e2g-input -v .:/var/tmp/e2g-output --env GHOSTFOLIO_ACCOUNT_ID=<some-acct-id>  --env DEBUG_LOGGING=true dickwolff/export-to-ghostfolio

> export-to-ghostfolio@1.0.0 watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Personal-Positions-2024-05-19.csv!
[i] Determined the file type to be of kind 'ibkr'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using IBKR converter
[e] An error ocurred while processing.
[e] Error details: Error: An error ocurred while parsing!
[e] Moving file to output..
[i] Stop container as usePolling is set to false..

Then when trying to import to Ghostfolio I get this error:

activities.0.currency is not valid
{
  "Positions for account Personal ...436 as of 04:48 PM ET, 2024/05/19": null,
  "": null
}
dickwolff commented 1 month ago

Hi there! Looks like a header line is being processed as a transaction. Could you maybe provide a (partial) export so I can reproduce this and make a fix? Thanks!

adrianbn commented 1 month ago

Here are the first few lines from one of the accounts:


"Positions for account Personal ...436 as of 04:48 PM ET, 2024/05/19","","","","","","","","","","","","","","","",""
"","","","","","","","","","","","","","","","",""
"Symbol","Description","Quantity","Price","Price Change %","Price Change $","Market Value","Day Change %","Day Change $","Cost Basis","Gain/Loss %","Gain/Loss $","Ratings","Reinvest Dividends?","Capital Gains?","% Of Account","Security Type"
"BNDX","VANGUARD TOTAL INTERNATIONAL BND ETF","330","$48.73","-0.27%","-$0.13","$1608.9","-0.27%","-$42.9","$1892.17","-15.03%","-$2844.27","--","No","--","4.06%","ETFs & Closed End Funds"

Then there are more entries like the BNDX one, and the account total at the end that looks like this:

"Account Total","--","--","--","--","--","$3961.6","0.2%","$80.17","$3413.54","13.55%","$462.64","--","--","--","--","--"

Exact dollar values and percentages are made up.

dickwolff commented 1 month ago

Thank you! I will take a look at it this week and get back to you!

dickwolff commented 1 month ago

@adrianbn The headers you posted don’t seem to match those of the sample export. The sample export is taken from the account history if I recall correctly (see the image in the readme). It seems like your export was taken from another screen and looks like it’s some kind of account position export (so the current status, not the transactions). Can you confirm this to me?

adrianbn commented 1 month ago

Hi @dickwolff,

Yes. I tried to import the positions because the history also was failing. This is what happens when I try to import the history:

> export-to-ghostfolio@1.0.0 watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Designated_Bene_Joint_XXXYYY_Transactions_20240520-165958.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol TDA TO DW&O TRANSFER (line 38)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..
npm notice
npm notice New minor version of npm available! 10.5.2 -> 10.8.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.0>
npm notice Run `npm install -g npm@10.8.0` to update!
npm notice

Then when I try to import the output file to Ghostfolio

activities.0.currency is not valid
{
  "Date": "05/16/2024 as of 05/15/2024",
  "Action": "Bank Interest",
  "Symbol": null,
  "Description": "BANK INT …XXX-YYYYYY TD BANK USA NA",
  "Quantity": null,
  "Price": null,
  "Fees & Comm": null,
  "Amount": "$3.11"
}

I think the line reporting isn't accurate (it may be skipping the headers and counting from 0?). I think this is the offending line

"11/07/2023","Internal Transfer","","TDA TO DW&O TRANSFER","","","","$7.06"

I tried with another of my Schwab accounts as well and the history export produces the following error:

> export-to-ghostfolio@1.0.0 watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Individual_XXXYYY_Transactions_20240520-170620.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol Tfr BANK OF AMERICA, N, XXXX YYYYYY ZZZ (line 30)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..
npm notice
npm notice New minor version of npm available! 10.5.2 -> 10.8.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.0>
npm notice Run `npm install -g npm@10.8.0` to update!
npm notice

I think the offending line is this

"04/08/2024","MoneyLink Transfer","","Tfr BANK OF AMERICA, N, XXXX YYYYYY ZZZ","","","","-$30.00"
dickwolff commented 1 month ago

Gotcha, thanks for the detailed analysis. One more thing to check: which version of the tool are you running?

adrianbn commented 1 month ago

Pulling from latest, -v reports 10.5.2.

On Mon, May 20, 2024 at 10:55 AM Dick Wolff @.***> wrote:

Gotcha, thanks for the detailed analysis. One more thing to check: which version of the tool are you running?

— Reply to this email directly, view it on GitHub https://github.com/dickwolff/Export-To-Ghostfolio/issues/75#issuecomment-2120928202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6NPRMIM5TOHGIPEDHMDR3ZDI2IRAVCNFSM6AAAAABH6VAM7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRQHEZDQMRQGI . You are receiving this because you were mentioned.Message ID: @.***>

dickwolff commented 1 month ago

That would be npm, but I meant the version of Export to Ghostfolio. The container version from Docker hub

adrianbn commented 1 month ago

sha256:06e91d68af18c6c853f4fa94593aaa0cf194cc8325d8b41331bdb89b6e741be9, 0.13.1 according to the labels; this is latest from last week or so.

dickwolff commented 1 month ago

Thanks! I'll probably have some time tonight to look into the issue and will get back to you when I have something for you to test 😅

dickwolff commented 1 month ago

I think the issue does relate to the "internal transfer" line, at least based on your information. Thanks again for the detailed infromation. I have added "transfer" to the ignore list, and that seems to work now.

Can you pull the 0.13.2-beta image and let me know if this resolved the issue? After your confirmation I will merge the change.

adrianbn commented 1 month ago

We're making progress but I am still having issues. One of my accounts has this problem:

> export-to-ghostfolio@0.13.2 watch
> tsx ./src/watcher.ts

[i] Watching /var/tmp/e2g-input..
[i] Found Designated_Bene_Joint_XXX436_Transactions_20240520-165958.csv!
[i] Determined the file type to be of kind 'schwab'.
[i] Starting Export to Ghostfolio v0.13.2
[i] Restored 0 ISIN-symbol pairs and 0 symbols from cache..
[i] Processing file using Schwab converter
[i] Read CSV file. Start processing..

[e] An error ocurred while trying to retrieve symbol TDA TRAN - NON-TAXABLE DIVIDENDS (CMF) (line 40)!

[e] An error ocurred while processing.
[e] Error details: BadRequestError: Missing required query parameter=q
[e] Moving file to output..
[i] Stop container as usePolling is set to false..

Offending line is this 

"11/07/2023","Non-Qualified Div","","TDA TRAN - NON-TAXABLE DIVIDENDS (CMF)","","","","$72.06"

This likely needs to be treated as a cash deposit I am guessing.

The other account shows no errors on the parsing, but it doesn't upload to ghostfolio. Pretty much every line errors out with:

activities.0.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
activities.1.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
activities.2.each value in type must be one of the following values: BUY, DIVIDEND, FEE, INTEREST, ITEM, LIABILITY, SELL
{
  "accountId": "a71e7a8c-dc55-4160-a461-4b6d9b752476",
  "comment": "",
  "fee": 0,
  "quantity": 29,
  "unitPrice": 0,
  "currency": "USD",
  "dataSource": "YAHOO",
  "date": "2024-05-17T00:00:00+00:00",
  "symbol": "ZZZZ"
}
dickwolff commented 1 month ago

The first error is a weird one. There are non-qualified dividends in the sample export as well, which I have got from another user when creating the Schwab converter. Those do have a symbol, like 09/29/2023,Non-Qualified Div,ARCC,ARES CAPITAL CORP,,,,$177.70 However your example does not seem to have a symbol. So that instance probably needs to be treated like a cash deposit, yes.

The last errors are due to the fact that the records don't have a type (buy, sell, etc) so that's probably a line that doesn't get converted correctly. What kind of record type do the lines have? So for 2023","Non-Qualified Div","",", the "Non-Qualified Div" part.

dickwolff commented 1 month ago

Sorry for the late check-in, I have been busy lately.

I added the non-qualified div as a cash position this morning. Could you try and run the tool again with the latest version, 0.13.2-beta? Thanks!

dickwolff commented 1 month ago

Merged the changes, closing issue due to inactivity. Please open a new issue should there be something else.