search

temna-net / navbot

Automatically exported from code.google.com/p/navbot
0 stars 0 forks source link

Automatically capture Market Data #13

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
The EVE interface only allows people to export items one at a time, and 
it's a fairly labour-intensive process. It'd be great if we could help 
with this - there are other tools out htere that capture the data stream 
sent when you simply view an item - that would be a start. We want to be 
careful not to annoy CCP, however!

Original issue reported on code.google.com by mark.oco...@gmail.com on 14 Oct 2007 at 9:09

GoogleCodeExporter commented 9 years ago 
Another option is to integrate the eve-metrics API...

Original comment by meisteryellow@gmail.com on 6 Jul 2009 at 11:17

GoogleCodeExporter commented 9 years ago 
There is a dll library apparently that opensource has that will auto grab the 
market 
using your cache files.

Someone already mentioned EVE-Metrics, it uses the cache to read the market 
data 
automatically.

Another one is called Traderoute discovery and analysis tool - Thread is 
New:-
http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1186277
Original:-
http://www.eveonline.com/ingameboard.asp?a=topic&threadID=1105136

This comment here is pretty interesting..
"The client is open source, and uses the libevecache library."

Original comment by senc...@gmail.com on 29 Sep 2009 at 3:11

GoogleCodeExporter commented 9 years ago 
wow would love to see eve-metrics and the cache based checking added to this 
anyoneknow 
if it can be done?

Original comment by ccha...@gmail.com on 30 Sep 2009 at 4:17

GoogleCodeExporter commented 9 years ago 
I think (I am at work right now so I can not check to make sure) that it is 
possible
already as is.

LibEveCache comes with a small tool called Dumper. This tool can create (or 
that's
how I interpret the documentation) Market Log type csv files by scraping the EVE
cache. It seems to me you could use this tool to just (manually for now) dump 
the
cache to standard market logs and feed those to NavBot.

Alternatively I also see an opening in the links sencnes forwarded. The tool he
speaks of uses clientside cache scraper tools that send the information to his
website. There it get's databased and processed to end up in a PHP IGB type 
page.

While I understand the writer of that tools desire to make a profit on it (you 
need
to buy an account with him using ISK) I am somewhat conflicted on that 
approach. I
guess I could be ok with it but still, it's borderline imho. Aside from that 
the tool
in question has the scraper clientside tools precompiled for windows and Mac 
OSX.
These clientside tools have a configuration file that allows you to change the
ipaddress of the server to send the scraped data too. The (uncompiled) linux 
client
sourcecode download is there too.

So I guess a second way of doing the cache thing is by using this clientside 
tool
from 'Traderoute Discovery and Analysis' and redirecting it to NavBot (also 
running
clientside). NavBot would 'just' have to be outfitted with a serverside 
connection to
receive the scraper client data.

I like this option too because it would remove the manual operation needed by 
dumper
(easy as it is to automate that). But more importantly it would allow for Mac 
OSX EVE
players to send data from their macs to a windows machine running NavBot.

That is how I run NavBot. I have 'shared' the marketlogs directory and NavBot 
uses it
via that share to generate the reports. I did have to change navbot slightly to 
make
the links relative instead of hardcoded to 'localhost' and even that can be 
avoided
by just manually changing the 'localhost' name in your IGB browser everytime you
click a link. But I digress.

When I get some time I might add one of these two options to the code and send 
it to
the author. Finding the time is the problem though.

Original comment by engel...@gmail.com on 5 Oct 2009 at 10:05

GoogleCodeExporter commented 9 years ago 
Manual or automatic it really doesn't matter, anyway is faster then selecting 
an 
item, saving the logs etc.

I use EVE-Metrics and it uses an semi-manual process, in which it literally 
just 
banks up the cache then once and a while sends the data to EVE-Metrics. However 
it 
does have a "Force Update" button.

What is really nice about the scaper is it captures everything very quickly. 
You 
could quiet easily collect the entire Jita market (And the whole region) 
without 
breaking a sweet. Manually click on each item and the cache just builds and 
builds, 
then you update and boom, you have every item both buy and sell orders for ALL 
"The 
Forge". Move to Sing and do the same thing. Maybe 20 minutes you'll have every 
item 
in The Forge and Sing and NavBot would spit out massive amounts of profitable 
routes.

I use NavBot now for some items I know, but automatic cache grabber would 
reveal 
items that you didn't even KNOW about. Adding this would make NAVBot one of the 
MOST 
DEMANDED programs I rekon, people always say "How do I make money with a 
Freighter?" 
If NAVBot had that, people would say, use contracts and NAVBot.

I'd easily put NAVBot up there with EVEMon if captured the market cache.

Original comment by senc...@gmail.com on 21 Oct 2009 at 10:43

GoogleCodeExporter commented 9 years ago 
I'm coding this ;).
Instead of making the dumper creating the cvs, i feed the data read from the 
.cache to 
NavBot.
It's better than dumping csv and then make NavBot read the csv since it's 
faster (and 
believe me.. with something like 700 files the cache decoder isn't so fast...).

Ah and hello everyone anyway, this is the first post here, i'm here to help you 
all ;)

Original comment by Smjert on 1 Feb 2010 at 12:13

GoogleCodeExporter commented 9 years ago 
Smerjt, such an update would be FANTASTIC! Thank-you for working on it :)

Original comment by Archidd...@gmail.com on 1 Feb 2010 at 4:29

GoogleCodeExporter commented 9 years ago 
>> I use NavBot now for some items I know, but automatic cache grabber would 
reveal 

>> items that you didn't even KNOW about. 

Could you explain, why it would reveal items that you didn't even know about?
Afaik item wouldn't appear in cache if you didn't view market details for it.

Original comment by icqk...@gmail.com on 1 Feb 2010 at 12:12

GoogleCodeExporter commented 9 years ago 
Awesome that would indeed make Navbot the Premium Trade Finder software...

Opensource, Local, and all Private (As in only you see the details you've feed 
into it) 
I can imagine once NavBot gets this Freighter Sales will climb as it will be 
extemely 
easy for people to go from region to region scanning down the market seeing 
what is 
what.

Original comment by senc...@gmail.com on 12 Mar 2010 at 9:42

GoogleCodeExporter commented 9 years ago 
Smjert did you finish the .chace -> navbot thing? can u share some code please?

Original comment by matija.g...@gmail.com on 22 Mar 2010 at 10:14

GoogleCodeExporter commented 9 years ago 
Would it be possible to integrate with multiple sources?

e.g. check
* local EVE cache
* eve-central
* eve metrics
and select the most up-to-date data?

Original comment by trent.ba...@gmail.com on 7 May 2010 at 5:19

GoogleCodeExporter commented 9 years ago 
I'm attempting to modify the EVE-Metrics uploader to not upload the data but 
save it
to a file instead, I will make it save them in the EVE log format, so NavBot can
parse them :)

Original comment by xeross.theelitist.net on 3 Jun 2010 at 3:06

GoogleCodeExporter commented 9 years ago 
Didn't manage to get it working properly, some weird ass error, might reattempt 
in
the future, but not in the mood for now.

Original comment by xeross.theelitist.net on 3 Jun 2010 at 4:28

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Any further development on this?

Original comment by adalo...@gmail.com on 4 Sep 2010 at 4:08

GoogleCodeExporter commented 9 years ago 
I can't seem to find where to download libevecache... just a blog...

Original comment by cabagepa...@gmail.com on 5 Sep 2010 at 6:31

GoogleCodeExporter commented 9 years ago 
Confirming comment #4.

LibEveCache (http://wiki.eve-id.net/Cache_Resources#libevecache_.28C.2B.2B.29)
can be used to export recently-viewed market data into CSV file(s) for 
importing into NavBot. NavBot issue #76 means that you can't import that 
information directly, but a workaround is possible. Just create a shell script 
that does

cat $INFILE | sort -r | uniq | grep -v "^$" > $OUTFILE

and that should fix the issue.

Original comment by serpenti...@gmail.com on 6 Sep 2010 at 5:35

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
For some reason the main.py file doesn't have a self.EVE or self.out

Also which main.py? I have 3 main.py's in the Python folder.

Original comment by adalo...@gmail.com on 7 Sep 2010 at 7:17

GoogleCodeExporter commented 9 years ago 
Great howto sencnes! It should be trivial to speed up NavBot's processing - 
it's never really been optimized as far as I know. If someone zips up a largish 
log set I'm happy to do this, or if another dev wants to then mail 
mark@codechart.com for a free account - codechart is a simple, ultra-clean .net 
profiler I've been working on ;-)

Original comment by mark.oco...@gmail.com on 7 Sep 2010 at 8:49

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I am with Adalonus. I can't find any self.exe or self.out in the main.py's...

Original comment by brandonj...@gmail.com on 7 Sep 2010 at 10:53

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
See, I was missing the AutoDumper part. Thanks a lot!

Original comment by adalo...@gmail.com on 8 Sep 2010 at 12:41

GoogleCodeExporter commented 9 years ago 
... :(
The black window pops up but then closes before anything can be done... i've 
tried reinstalling and following the steps again...

why would the window close?

Original comment by cabagepa...@gmail.com on 8 Sep 2010 at 12:52

GoogleCodeExporter commented 9 years ago 
from the py command line (not opening it from the main.pv file), how can i open 
the main.pv? ~maybe i can find a way to make this work...

Original comment by cabagepa...@gmail.com on 8 Sep 2010 at 12:54

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
I'm getting the same problem. The black window opens and quickly closes. It 
fails to export anything from the market

Original comment by adalo...@gmail.com on 8 Sep 2010 at 12:58

GoogleCodeExporter commented 9 years ago 
damnit... so close....

anyone else besides us three with this issue?

Original comment by brandonj...@gmail.com on 8 Sep 2010 at 2:38

GoogleCodeExporter commented 9 years ago 
I'm really sorry guys, I missed something. I decided to launch a VM and install 
it on there to see what I needed.. Here it is the final full set, fully tested 
on a fresh image, and found working...

These are the EXACT files I used :)
Sorry no Win64 yet, When you install Python 64, Reverence doesn't detect it.

Download and install Python 2.7 (Yeah I upgraded because I wanted to)
http://www.python.org/ftp/python/2.7/python-2.7.msi
Install it

Download Reverence 2.7
http://github.com/downloads/ntt/reverence/reverence-1.1.1.win32-py2.7.exe
Install it

Download Win32con (This is the part I was missing, and why when I retested it 
still worked, only a new fresh image did I get the error)
http://downloads.sourceforge.net/project/pywin32/pywin32/Build%20214/pywin32-214
.win32-py2.7.exe?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fpywin32%2Ffiles%2F&
ts=1283915478&use_mirror=superb-sea2
Install it

Download Autodumper
http://dl.eve-files.com/media/corp/mazen/autodumper.zip
Install it

Edit main.py in the "Autodumper" folder
self.eve to match your Eve location
For me The whole line is - self.eve - blue.EVE("E:\\Games\\EVE")
self.out to match the marketlogs location
For me The whole line is - self.out = 
"C:\\Users\\SencneS\\Documents\\EVE\\Logs\\Marketlogs"

Save & Close main.py
Right click on main.py "Open" (Or double click)

A black window will appear, as you scan the market it will save the logs.

Sorry about that guys like I said I had been using it for a while and I forgot 
about that one small requirement.

Tips - Check EVE Location, I had accidentally made mine D: instead of E: for my 
EVE Folder and it wouldn't open for me. So make sure the paths are correct.

Feel free to donate some ISK to the guys who created Reverence (Entity), 
Autodumper (Mazen Alane), NavBot (Whoever is still playing EVE) lol. And of 
cause me who found the magic connections to link them all together (SencneS) 
/Shameless plug for ISK :)

Original comment by senc...@gmail.com on 8 Sep 2010 at 3:31

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
.... God i feel horrible saying this cause i know you are trying (i wish i 
could program myself)...
but now the black screen is up and running... (ill attach a screen shot of it 
in test run)...
but nothing is actually created in the folder...

Original comment by brandonj...@gmail.com on 8 Sep 2010 at 11:16

Attachments:

GoogleCodeExporter commented 9 years ago 
...im gonna try uninstalling it and installing the 2.6's ...

Original comment by brandonj...@gmail.com on 9 Sep 2010 at 1:50

GoogleCodeExporter commented 9 years ago 
nope. 2.6 black window opens then quickly closes even with the 2.6 version of 
win32con (found off google search)...

Original comment by brandonj...@gmail.com on 9 Sep 2010 at 2:08

GoogleCodeExporter commented 9 years ago 
I see your problem here. The screenshot shows the save location as

C:\Userwinks\Documents\EVE\Logs\Marketlogs\

Since you're using Vists I know that's wrong. Go to the Main.py you edited and 
make sure the self.out line reads...

self.out = "C:\\Users\\winks\\Documents\\EVE\\Logs\\Marketlogs"

I suspect you may have only one \ in there when it needs two.

This screenshot actually reveals a lot, it shows you have Python, Reverence, 
win32con and autodumper working, but the location it's told to save the logs is 
a little incorrect. I'd go back to whatever versions you used to get this 
screenshot 2.7.

Original comment by senc...@gmail.com on 9 Sep 2010 at 1:24

GoogleCodeExporter commented 9 years ago 
You guys are great. Thanks for all the help.

Now if only there were a way to do inter-regional prices (I suppose jump 
clones, but i can't install them yet)

Original comment by adalo...@gmail.com on 9 Sep 2010 at 8:00

GoogleCodeExporter commented 9 years ago 
Thanks alot sencnes! I'll see what i can do to donate once i get a few mill 
(just started account)

Original comment by brandonj...@gmail.com on 10 Sep 2010 at 2:41

GoogleCodeExporter commented 9 years ago 
I ported the autodumper to linux, here goes. See README.txt

Send ISK to Cindril if you like it :-)

Original comment by georgpsw...@googlemail.com on 28 Nov 2010 at 8:24

Attachments:

GoogleCodeExporter commented 9 years ago 
I read the whole post and have no idea what you guys are talking about. I am 
intrested however in the fact that eve central has a page that when viewed on 
IGB and trusted it loads the market automatically and goes threw a list of 
items and exports the data automatically to eve-centrals host client extention. 
So my unskilled in coding mind wonders, if they can do that, can we come up 
with a client extention, like Navbot, that uses a local host website, separet 
as to be able to run coincide with Navbot, that when you load you make an item 
list, probably on set up/start up in host client, Then when you open the page 
it starts looking at those items over and over while you auto pilot to your 
next destination. Like I think you guys said if it could scan the cash data and 
turn it into maker data that Navbot can understand that would be awsome.

My plan is to make a "traveling salesman problem" route for an alt to run 
around in a shuttle while I run around on my main in a freighter. Feeding 
myself data and running freight concerently.

If this is possible or you are up to the challenge I would love some help with 
this pretty please. I'm sick of eve-centrals miss information.

Original comment by Visr...@ymail.com on 10 Feb 2011 at 6:51

GoogleCodeExporter commented 9 years ago 
The link for downloading Reverance is outdated. here is the new link 
https://github.com/ntt/reverence/

Original comment by brandonj...@gmail.com on 5 Jun 2011 at 2:29

GoogleCodeExporter commented 9 years ago 
I'm loving this program, thanks for putting it all together.  But I'm a lazy 
one, and wonder if this can be taken a step further.  Is there anyway to 
automatically update the cache with every market order without opening every 
category in the market.  Even something as simple as being able to press the 
down arrow all the way through would be helpful.  Thanks for what's been done 
already though, it works perfect.

Original comment by PDu...@gmail.com on 10 Jun 2011 at 4:54

GoogleCodeExporter commented 9 years ago 
You can use javascript + in-game browser to open market window screens.

e.g. http://eve-marketdata.com/update_market.php
or http://eve-central.com/home/upload_suggest.html

I'm sure navBot can do something similar.

Original comment by serpenti...@gmail.com on 10 Jun 2011 at 5:47

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
with python 2.7.1 and reverence update for incarna (currently up for binary 
download) your cache to eve export format scripts should be fine. mine are :)

Original comment by hitchhik...@gmail.com on 14 Jul 2011 at 5:54

GoogleCodeExporter commented 9 years ago 
Is anyone having issues getting this to work on the Crucible?

Original comment by brandonj...@gmail.com on 2 Jan 2012 at 5:48

GoogleCodeExporter commented 9 years ago 
I came back to Eve and was hopping this add on would be able to work like 
everyone seems to want it to work. Auto export data and show me trade routs.

Original comment by Visr...@gmail.com on 6 Jan 2012 at 8:33

GoogleCodeExporter commented 9 years ago 
Ok, I've done everything you guys said and I edited autodumper like you guys 
said but the box just closes. I don't know what I'm doing wrong and I'm not a 
programer. Just trying to make my trading easier. Navbot works great when I 
export all my data myself but automating that would be awesome.

Using windows vista ultimate 64-bit.

Would be really nice if there was just one file to download and install the 
file to auto export and give me my trade routs on the IGB. I'm not smart but it 
doesn't take a genius to realize this is a gold mine.

Original comment by Visr...@gmail.com on 6 Jan 2012 at 9:12

GoogleCodeExporter commented 9 years ago 
Its an issue with crucible according to another forum... but if there's a way 
to get around the new changes CCP made, idk... (not a programmer either)

Original comment by brandonj...@gmail.com on 6 Jan 2012 at 12:50

GoogleCodeExporter commented 9 years ago 
AutoDumper doens't seem to be exporting to a format that NavBot understands.

Original comment by dboil...@gmail.com on 15 Jan 2012 at 11:43