Instruction for running the scripts

[HenryLeongStat]

Currently, the script is written as CLI (command line interface). If you want to try the script, open terminal, and then run the script like the followings: python arg_clawer.py -url http://xxxxx

For example:

python arg_clawer.py -url https://cehi.rice.edu/about/staff

The output will be like:

......
 Marie Lynn Miranda PhD,
 Director, Howard R. Hughes Provost and Professor

 Dr. Marie Lynn Miranda is the Howard R. Hughes Provost and Professor of Statistics at Rice University, holds appointments as adjunct professor in the Departments of Pediatrics at the University of Michigan and the Baylor College of Medicine, and serves as the founding Director of CEHI. Dr. Miranda's educational background is rooted in economic and mathematical modeling; her professional experiences integrate environmental health sciences with sound social policies. She has taught courses and conducted research on children's environmental health, with a particular emphasis on reproductive and developmental toxicants, childhood lead exposure, and allergen and asthma triggers. Dr. Miranda has applied spatial analytic approaches to a wide range of environmental issues. She also has extensive experience running training, research translation, and outreach programs, especially as they relate to disadvantaged populations. Dr. Miranda has an active research portfolio, with funding from the USEPA, NIH, CDC, the National Association of Chronic Disease Directors, the USDA, the State of North Carolina, the Mary Duke Biddle Foundation, and The Duke Endowment. She maintains a deep and abiding personal and professional interest in environmental and social justice.
 Attention:Rice University – BRC Children’s Environmental Health Initiative (CEHI), MS-460 6100 Main St Houston, Texas 77005

 Rebecca Anthopolos,
 Statistician

 Rebecca Anthopolos serves as a statistical  analyst at CEHI. Her research and statistical analysis at CEHI focuses  on determining social and environmental factors contributing to  disparities in birth outcomes in the South. She completed her Master's  in Economics from Duke University in May 2007 and also holds a Bachelor  of Arts in International Relations with an emphasis on international  development from Connecticut College.
 ra42@rice.edu
 Attention:Rice University – BRC
Children’s Environmental Health Initiative (CEHI), MS-460
6100 Main St
Houston, Texas 77005

 Kathleen Bergen,
 Associate Research Scientist

 Dr. Kathleen Bergen is an Associate Research Scientist in SNRE holding a part-time appointment with CEHI.  Her background and research is in ecology, biodiversity, coupled human-natural systems, information science and remote sensing/GIS. She is involved in CEHI work focusing on environment and human health and well-being that draws on these and related domains. She also assists in grant-writing and publication and curricular preparation.
 kbergen@umich.edu
 734-678-0275
 Attention: CEHI
3506 Dana Building, 440 Church St
Ann Arbor, MI 48109

 Mercedes Bravo,
 Post-doctoral Research Fellow

 Mercedes Bravo serves as a Post-doctoral Research Fellow at CEHI. Her research has focused on developing new methods of estimating air pollution exposure and investigating links between exposure and health outcomes in understudied populations in the U.S. and abroad. Mercedes received her Ph.D. in Environmental Science (2014) from Yale University in New Haven, Connecticut, and also has a Bachelor of Science degree in Environmental Science/Chemistry from the University of North Carolina at Chapel Hill.

 mbravo@med.umich.edu
 (734) 764-4584
 Attention: CEHI
4040 Dana Building, 440 Church St.
Ann Arbor, MI 48109

 Brandi Gih,
 GIS Technician

 Brandi Gih serves as a GIS Technician. She holds a Bachelor of Arts in Geography and American Studies as well as a Masters of Public Policy from the University of Northern Iowa. Her work at CEHI includes data management and technical support for GIS research projects. Her research interests concern the application of GIS data and techniques for environmental health analysis.

 bgih@rice.edu
 (734) 763-2003
 Attention: CEHI
4044 Dana Building, 440 Church St
Ann Arbor, MI 48109

 Jocelyn Hwang,
 GIS Technician

 Jocelyn Hwang serves as a GIS Technician at CEHI. Her responsibilities include supporting training efforts and providing technical support for GIS projects. She is interested in using spatial analysis to understand and improve environmental health, especially in cases of health inequity. Jocelyn received her Masters of Health Science as well as Bachelors of Science in Environmental Engineering from Johns Hopkins University.
 Jocelyn.Hwang@rice.edu
 (713) 348-8143
 Rice University – BRC Children’s Environmental Health Initiative (CEHI), MS-460 6100 Main St Houston, Texas 77005

 Hien Le,
 Business Systems Analyst

 Hien Le is a Business Systems Analyst who works on projects from the Children’s Environmental Health Initiative and the Kinder Institute for Urban Research.  He liaises with Rice Office of Information Technology on items concerning the usage, maintenance, and development of the Virtual Research Data Environment (VRDE).  He also assists in the acquisition, curation, and management of datasets.  Hien received a Bachelor of Engineering degree in Electrical Engineering from McGill University.
 Hien.Le@rice.edu
 (713) 348-8241
 Attention:
Rice University – BRC
Children’s Environmental Health Initiative (CEHI), MS-460
6100 Main St
Houston, Texas 77005

 Ruiyang Li,
 GIS Technical Team Leader

 Ruiyang Li is a research area specialist associate at CEHI, a certified Geographic Information Systems Professional (GISP). He graduated as a Master student from School of Natural Resources & Environment, University of Michigan in 2014. He is interested in the application of spatial analysis in public health at CEHI. He is excited to be a part of CEHI team; and wants to contribute in improving the health quality of various communities.
 ruiyang.li@rice.edu
 (713) 348-8263
 Rice University – BRC Children’s Environmental Health Initiative (CEHI), MS-460 6100 Main St Houston, Texas 77005

 Brian Neelon,
 Associate in Research

 Dr. Brian Neelon is a CEHI statistician providing statistical support  for grant applications, data analysis, and the development of new  statistical methodology related to environmental and child health. He  received his doctorate in biostatistics from the University of North  Carolina at Chapel Hill, and arrived at CEHI after completing a two-year  postdoctoral fellowship at the Harvard Department of Health Care  Policy. His research interests include Bayesian methodology,  longitudinal data analysis, and latent trajectory models. For more  information, please visit Dr. Neelon's webpage at http://www.duke.edu/~neelo003/.
 neelo003@duke.edu
 (919) 613-8736
 Levine Science Research Center         308 Research DriveRoom A121         Durham, NC 27708

 Claire Osgood,
 Data Manager

 Claire Osgood is CEHI's Data Manager. She graduated from Duke University  with a double major in Computer Science and Economics. She is involved  in the acquisition, processing, and maintenance of data used by CEHI,  with an emphasis on SAS programming. Her goals are to use data and her  programming skills to help affect positive change in our community.
 claire@duke.edu
 (919) 668-9314
 310 Trent DriveRoom 208 Trent HallDurham 27710

 Joshua Tootoo,
 Director of Training and Geospatial Sciences

 Joshua Tootoo, serves as CEHI's Director of Training and Geospatial Sciences His work and responsibilities involve the coordination of CEHI's GIS capacity building efforts and the management of the group's geospatial projects. A certified Geographic Information Systems Professional (GISP), he completed his Masters in Environmental Planning from the University of Rhode Island and also holds dual Bachelor of Science degrees in Botany and Forest Ecology from North Carolina State University.

 jltootoo@rice.edu
 (713)348-8244
 Attention:Rice University – BRC Children’s Environmental Health Initiative (CEHI), MS-460 6100 Main St Houston, Texas 77005
...

[HenryLeongStat]

For pic_crawler.py, it is used to get the url of the pictures in a website. Example:

manchongleong@Mande-MacBook-Pro:~/Documents/GitHub/CEHI-Web-crawler-for-residential-history$ python pic_clawer.py -url https://cehi.rice.edu/about/staff
/sites/g/files/bxs176/themes/site/images/rice-logo.jpg
https://cehi.rice.edu/sites/g/files/bxs176/themes/site/logo.png
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/mlm_old_0.jpg?itok=ElFCt4xx
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/RAnthopolos_0.jpg?itok=hiURPDSW
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/kbergen_photo.jpg?itok=jEXkkwOM
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/MercedesBravo.jpg?itok=Vj9gnlz1
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/BG_final_0_0.jpg?itok=db9wLgZH
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/Jocelyn_0.jpg?itok=ibbpN9YR
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/Hien_Bio_Sketch_Picture_0.jpg?itok=oe-IIDAQ
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/Yang_0.jpg?itok=Xu9IIj8z
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/brian_0.jpg?itok=aRvSr0Zo
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/claire_0.jpg?itok=aDALRHvT
https://cehi.rice.edu/sites/g/files/bxs176/f/styles/sdefault/public/Joshua-bw-nowb.jpg?itok=9zeGulpB

[HenryLeongStat]

Links are somethings elemental when doing the reclusive web crawling.