search

owocki / pytrader

cryptocurrency trading robot
MIT License
1.93k stars 433 forks source link

Investigate Adding social semantics #8

Open owocki opened 8 years ago

owocki commented 8 years ago

I have not incorporated social semantics in this yet, but I'm sure there's some signal there. I'd bet reddit, bitcointalk, and twitter would be good places to start

NDuma commented 8 years ago

Yes, I was referring, more directly, to starting with screening the chatrooms available through the exchanges API's. Poloniex has this available; and, taking advantage of how many times something is shared and where it is referenced might be an indicator of it's importance, or it's 'popularity' for another reason - finding the reason; or even taking advantage of pumps. Even vetting influencers.

owocki commented 8 years ago

@NDuma noodling on this a bit more. the first step would simply be to start collecting the data.

i'm thinking that each currency pair has a set of keywords (ETH, Etherueum, vitalik buterin) for ETH as an example. One could plausibly hit the API of a set of online chat rooms (polo trollbox, bitcointalk, r/ethtrader, etc), and simply store the text of any keyword matches in a time stamped model to start.

Once the data is stored in the model, we could think about feeding this into a classifier or a NN, and could maybe do some sentiment analysis of the comments to provide additional signal.

I wonder if this should be a priority over another strategy, say, for example, trying to minimize the impact of fees on the trading bot.

NDuma commented 8 years ago

Agreed, @owocki; a synonym and keyword list with associations would be needed; and, fairly easy to produce an initial set (Andrew in the room invited has one made already; but, not yet shared). Later, other words might be found to have -/+ correlations when used in context or by author (ie: bank, acquired, start-up, new platform - semantic data analysis might show something interesting).

Signal is the highest priority; as, it seems, without having actually looked at anything other than the description given (1 BTC traded w/ 2.5 in fees), the bot is over trading with not enough profit in each trade; meaning, it needs stronger signals for larger % gains in each trade to make up for the 0.15 - 0.25% fee. Is a list of all trades made available?

The reason I suggest actual chat boxes (PoloniBox.com keeps a public record available; but, has experienced some technical issues lately - the feed is available through the API anyhow) is because of the time relevant information being updated in a stream. StockTwits has been around for a while and the question is how much social is needed to push any particular amount of volume.

More sources would include lists of all the Official feeds from blogs to the organizations users / devs feeds to be monitored for updates. Back-testing for semantic correlations & other media outlets: ie: NYT / theMerkle.

Before bots, the market's all about psychology; anything can go to 1 billion if the people buying believe it should or could.

NDuma commented 8 years ago

Also reference #13 in regards to indicators people use and mention.

NDuma commented 8 years ago

Also #20

jeff-hykin commented 8 years ago

This may be a more appropriate place for my comment on #20

Have you looked at all into using Social Mention or Google Alerts for social sentiment analysis?

Sifting and reacting to viral bursts of social approval could provide large gains with minimal trading.

Update: I wrote a simple HTML scraper in python that grabs the top four scores from Social Mention screen shot 2016-03-30 at 2 32 03 am Here’s a link to the code, at minimum it’s a proof of concept

owocki commented 8 years ago

Hey @jeff-hykin that's certainly an interesting idea. I wasn't planning to have this repo be used to react to current events ( under the assumption that anything real time would require much more of a focus on performance ), but it's worth giving it a try. I wish this was active this week when the NYTimes Ethereum article came out, that would have been a great test case.

rmendes900 commented 8 years ago

Also, we have reditt/ and specifc Crypto channels and web sites like Whaclub, Stakepool and the Telegrams. They have important feed for the Bots. At Crypto Market, the info that rules came from the Devs... This is very important!!

2016-03-30 10:28 GMT-03:00 Kevin Owocki notifications@github.com:

Hey @jeff-hykin https://github.com/jeff-hykin that's certainly an interesting idea. I wasn't planning to have this repo be used to react to current events ( under the assumption that anything real time would require much more of a focus on performance ), but it's worth giving it a try. I wish this was active this week when the NYTimes Ethereum article came out, that would have been a great test case.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/owocki/pytrader/issues/8#issuecomment-203432381

jeff-hykin commented 8 years ago

Okay @owocki So you’re saying pytrader operates in significantly less than real time? I apologize if this has a fairly obvious answer, I have a background in machine learning and AI but relative to this project I am inexperienced.

Do you think the information would be useful for non-real time projections? Ex: TSLA stock has risen from ~$170 gradually up to ~$230 (with a cap ~$280) since their mention of the model 3. This is likely in correlation with public option and media coverage. In fact their stock is likely to rocket tomorrow with the actual model 3 event.

What did you have in mind for adding social semantics?

owocki commented 8 years ago

@jeff-hykin Hi Jeff, currently, pytrader operates by using classifiers and NNs to predict price movements based upon the last n price movements of granularity g. The trade.py loop sleeps for 95% of the time until it has an opportunity to make another evaluation like this.

I suppose what I'm saying is that no effort has been put into detecting trends on social media. If we were to add trend detection into the system, I think we'd have to find a way to make the trade.py engine responsive to rapidly incoming information from social media. There is certainly a role for social data to play, I just won't know what until we come up with some theory and successfully backtest it!

TLDR - Nothing social added to system yet, but it seems like it could have some promise.

jeff-hykin commented 8 years ago

@owocki Got it!

TLDR - I’ll start theorizing / testing independently and let you know if I find something interesting/practical

I’m personally interested in the social media aspect (even before PyTrader) so I may try to work on creating a successful way to integrate it. Currently my approach would be to have an external module sorting and finding patterns in social media. When it thinks it found something significant it would notify or startup the trade.py loop with some given parameters. It’ll have some method for combining its confidence with that of PyTrader. This is all (obviously) pretty theoretical, but let me know if this is blatantly incompatible with PyTrader.

Last night I ordered a Raspberry Pi 3 so I can run some Python scrapers 24/7 and build up a database (something I’ve been meaning to do for awhile). This should be a good first step for any direction with social semantics.

Update:(4/13/2016) for many accuracy reasons it looks like SocialMention is not a good bet.

Also my raspberry Pi 3 came so I will be experimenting with data scraping with it in the next few weeks

_

Goals: Collect Data with Raspberry Pi Organize & Save Data Analyze Data Get Market Data Machine learning: Attempt to Predict Market Up and Down Spikes with Social Media Data Test Accuracy Analyze possibility of Overfitting Discover or Fail to Discover Profitability If Profitable, integrate with PyTrader

Done: Scrape Data Get Raspberry Pi

owocki commented 8 years ago

TLDR - I’ll start theorizing / testing independently and let you know if I find something interesting/practical

Awesome. I'm putting in some cycles this weekend so maybe I'll catch you on slack and we can have a jam session (I'm in MST)

I’m personally interested in the social media aspect (even before PyTrader) so I may try to work on creating a successful way to integrate it. Currently my approach would be to have an external module sorting and finding patterns in social media. When it thinks it found something significant it would notify or startup the trade.py loop with some given parameters. It’ll have some method for combining its confidence with that of PyTrader. This is all (obviously) pretty theoretical, but let me know if this is blatantly incompatible with PyTrader.

You might try creating a separate django app ( ./manage.py startapp) with the models/logic. Once the basic structure / social data is available, you and I can work together on how it integrates into the NN/trader engine.

Last night I ordered a Raspberry Pi 3 so I can run some Python scrapers 24/7 and build up a database (something I’ve been meaning to do for awhile). This should be a good first step for any direction with social semantics.

Coolio - The Pi 3 seems really powerful :) I have some Pi's around my house for various home automation projects, but I'm mostly a cloud guy when it comes to this kind of project. A few of us on the pytrader slack have set up servers at digitalocean. I was thinking of setting up a shared reference implenetation ( https://github.com/owocki/pytrader/issues/24 ) that will always have up to date prices (and social data if we get there ) that we can all pull into our individual trading instances.

jeff-hykin commented 8 years ago

TLDR: I’m still learning

Been busy with school the past week, but it ends in 3 weeks so that’ll help

I got the Pi but haven’t done much with it other than get the OS running. I downloaded Two Scoops of Django which looked really helpful and correctly referred me to "Writing your first Django app, part 1” because I know nothing about it haha, so it might be a bit before I’m setting up a separate Django app on my own.

I also recently realized I don’t even know many of the basics of Python, despite writing many Python files. I never took the time to learn it from the ground up so I’ve been missing a lot of the most basic details. (I had no idea what pep8 was or CPython or anything similar) Currently I found a great app by SoloLearn called “Learn Python” and have been moving through it quickly. I’ll likely be done with it in a few weeks and will then be programming stuff for the Pi 3 as well as learning Django.

owocki commented 8 years ago

v0.1 of this, containing twitter data => https://github.com/owocki/pytrader/pull/75

NDuma commented 8 years ago

@owocki I noticed reddit has been done / added as well. Should we make a list and prioritize them somehow? I think weightings might be relevant per individual on something like StockTwits pending their historical & follower size etc... Also, the exchanges have API's which tie into their chat. . Analyzing news outlets might be a bit more challenging in discerning the content of the sentiment.

owocki commented 8 years ago

Should we make a list and prioritize them somehow?

Yeah, Im thinking that we got the main ones, but I'd be happy to shown wrong.

I think weightings might be relevant per individual on something like StockTwits pending their historical & follower size etc...

Thats a good point. I'll noodle on how that gets added to the model. Maybe adding a amplifier column to the model added in the work in #75.

Also, the exchanges have API's which tie into their chat.

I'd be curious if/how trollboxes provide any signal. Its certainly worth testing.

Analyzing news outlets might be a bit more challenging in discerning the content of the sentiment.

Yeah, agree. More objective. But could be some signal in the volume.

NDuma commented 8 years ago

we got the main ones

Over time more detail & fleshing which have weighted value; Bayes might be helpful.

More objective. But could be some signal in the volume.

or even the Alexa traffic ratings of the site; or, if the site has view / share #'s published

if/how trollboxes provide any signal.

We'd had someone analyzing polo's; although I'm not sure what else they were doing with it. Might be good to find which BTCTalk articles are actually circulating & being cited (other places as well) - so they're not all weighted the same. ++ Pumps in those boxes do happen; even with mods.

Maybe adding a amplifier column to the model added in the work in #75.

Think that's a great idea; must've skipped over that issue thread. There'll be more to add; copula of multivariate correlated distributions / covariate matrices ... ... Apologies; in advance - 'citation' is condensed: Following some of ICITE's work and In-Q-Tel programs with IARPA like Open Source Indicators Program and the "Good Judgement Project" via the ACE program - Social indicators to commodity prices can predict events ahead of time across the globe if done correctly; social sentiments might have negative correlations in specific scenarios / contexts.

gitcoinbot commented 6 years ago

This issue now has a funding of 0.02 ETH (16.02 USD) attached to it.

owocki commented 6 years ago

@gitcoinbot test

gitcoinbot commented 6 years ago

@owocki. :wave: thanks for the atMention, but you need to install @gitcoinbot on this repo for me to be able to respond. More details in the documentation.

:v: @gitcoinbot

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.1 ETH (69.21 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.1 ETH (69.21 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.1 ETH (69.21 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.1 ETH (69.21 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.03 ETH (20.76 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.03 ETH (20.76 USD @ $692.1/ETH) attached to it.

darkdarkdragon commented 6 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.03 ETH (20.76 USD @ $692.1/ETH) attached to it.