search

xiaoleiy / letlink-crawler

Automatically exported from code.google.com/p/letlink-crawler
0 stars 0 forks source link

Web Crawler #1

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Web Server: Tomcat
OS: Ubuntu Linux server
Techs: jQuery, JS, Ajax, css, monitoring tools
Additional struts action classes should also be developed to react to the web 
client.

Original issue reported on code.google.com by yuleib...@gmail.com on 5 Jul 2011 at 3:22

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Modules & Functions:
1, CURD operations on: domain, domainConfig, seedlist, cron (AKA scheduler in 
Netarchive), crawler excluded urls (AKA global traps), crawlers (AKA harvester 
in Netarchive)
2, Crawlers/Job Management: Crawlers listing and operations, Jobs listing, 
running status, statistics, progress
3, Image Viewer: Images listing (by page), operations of view(in bunch), 
search, order manually, delete, introduce to project-phase-1.
4, System Monitor: 
    1) Web server(dispatcher) monitor, Distributed crawlers & OS monitor,  Distributed DBs monitor, FTP server monitor.
    2) running logs, 
    3) Statistics data: processed webpages (url), to-process webpages(url), crawled images, crawling speed(kb/s, images/s), datasize of crawled images (MB), chart for historical crawling speed.
    4) Operations: start/stop distributed crawlers

Original comment by yuleib...@gmail.com on 5 Jul 2011 at 7:29

GoogleCodeExporter commented 9 years ago 
<<TODO>>
1, How could I validate administrator's login by connecting to project phase-1? 
by DB connection, or interface?
2, Make the login.jsp page invisible to clients.
3, Encrypt the password cookie in login.jsp or in Login servlet

Original comment by yuleib...@gmail.com on 7 Jul 2011 at 3:46

GoogleCodeExporter commented 9 years ago 
TODO:
1, 
子域名如何解析到具体的页面上？如，http://idealab.javaeye.com解
析为http://www.javaeye.com/user?uid=idealab
2, Why the context listener definition *MUST* be after servlets definition and 
before filter definition???
3, Ajax跨域访问问题

Original comment by yuleib...@gmail.com on 20 Jul 2011 at 2:28