search

eprbell / dali-rp2

DaLI (Data Loader Interface) is a data loader and input generator for RP2 (https://pypi.org/project/rp2), the privacy-focused, free, open-source cryptocurrency tax calculator: DaLI removes the need to manually prepare RP2 input files. Just like RP2, DaLI is also free, open-source and it prioritizes user privacy.
https://pypi.org/project/dali-rp2/
Apache License 2.0
65 stars 43 forks source link

Proposal for HTML parser #40

Closed macanudo527 closed 2 years ago

macanudo527 commented 2 years ago

I have a one or two exchanges, one being gate.io, which does not allow you to download quant trades as csv or via REST. They do display these trades in html form, however. Is it possible to create HTML plugins?

It would basically be very similar to the CSV plugins, just read in HTML instead of CSV. It would be nice to use the BeautifulSoup package, which is a pretty easy to use html parsing tool instead of Python's html.parser.

The idea would be to download the html of the trades then feed it into dali. Any thoughts?

eprbell commented 2 years ago

The problem with HTML scraping is that it's very fragile: as soon as a webmaster changes slightly the HTML code the scraping code can break. It's hard to build robust code based on HTML parsing. Can you share more about this gate.io limitation: what are quant trades and are they the only thing that is not available through their REST API?

macanudo527 commented 2 years ago

The quant trades are basically trades made by bots, 'quant' being short for quantitative. Either the trade is based on some kind of formula of market indicators or it's a simple grid trading bot or infinity grid bot or something like that. Basically, it is making 10~20 trades a day on your behalf, so it would help to automate that tracking.

The quant trades seem to be the main thing that is missing. I guess the people that use the bots don't do taxes? I have no idea why they wouldn't include this, and maybe they will in the future. I also use another exchange, Pionex, that doesn't have an API at all. They do allow you to request a csv of transactions but you have to email them, and even then you can only request it 3 times a quarter. It would be nice to just be able to parse this myself.

I can always just duct tape something together that will parse it into csv and then feed it through Dali, or simply use it as an unofficial plugin, but I thought I would ask.

eprbell commented 2 years ago

Thanks for the info: that's interesting. Strange that they wouldn't cover those transactions via REST or CSV... I think the strategy you outlined is probably the best: have a side script outside of DaLI that generates a CSV for the manual plugin from the native HTML.

eprbell commented 2 years ago

One more question: have you tried contacting their customer support about the missing REST API functionality (if they have a Twitter handle, maybe do it directly on social media)? Perhaps you can put pressure on them to add the missing part. I imagine you're not their only customer hitting this problem.

macanudo527 commented 2 years ago

I did try contacting them and didn't get a solid response. I just kept getting passed off to other people. It's not such a big concern of mine since everything I have on there is hobby mining income, which will have a $0 basis price anyway since I'm probably not going to bother writing off my equipment / energy costs.

For the Pionex exchange, I can just use the CSV they give me and hope they implement an API in the future so I can get more immediate data.

macanudo527 commented 2 years ago

I'll work out something on my own. Thanks for your input.