ros-infrastructure / rosdoc_lite

A light-weight version of rosdoc that does not rely on ROS infrastructure for crawling packages.
10 stars 31 forks source link

rosdoc jobs are creating many GBs of data in jenkins WORKSPACE #7

Closed tfoote closed 11 years ago

tfoote commented 11 years ago

Below is a snippet of the accumulated workspaces on a single executor

...
7.3M    doc-fuerte-rospkg-precise-amd64
36M doc-fuerte-ros-precise-amd64
136M    doc-fuerte-ros_realtime-precise-amd64
48M doc-fuerte-rosserial_android-precise-amd64
60M doc-fuerte-rosserial_java-precise-amd64
8.1M    doc-fuerte-rosserial-precise-amd64
32M doc-fuerte-ros_tutorials
53M doc-fuerte-ros_tutorials-precise-amd64
53M doc-fuerte-rovio
15M doc-fuerte-rqt
130M    doc-fuerte-rx-precise-amd64
94M doc-fuerte-scan_tools
226M    doc-fuerte-schunk_modular_robotics-precise-amd64
103M    doc-fuerte-schunk_robots
83M doc-fuerte-schunk_simulation
61M doc-fuerte-schunk_simulation-precise-amd64
3.9M    doc-fuerte-serial_communication-precise-amd64
485M    doc-fuerte-shadow_robot-precise-amd64
39M doc-fuerte-simple_object_capture
37M doc-fuerte-simple_object_capture-precise-amd64
238M    doc-fuerte-simulator_gazebo
1.3G    doc-fuerte-simulator_gazebo-precise-amd64
32M doc-fuerte-slam_gmapping-precise-amd64
61M doc-fuerte-slam_karto-precise-amd64
13M doc-fuerte-sound_drivers-precise-amd64
332K    doc-fuerte-sql_database-precise-amd64
16M doc-fuerte-srs_common-precise-amd64
406M    doc-fuerte-sr_visualization
756K    doc-fuerte-sr_visualization-precise-amd64
53M doc-fuerte-stage-precise-amd64
24M doc-fuerte-std_msgs
22M doc-fuerte-std_msgs-precise-amd64
121M    doc-fuerte-symbolic_planning-precise-amd64
251M    doc-fuerte-tabletop_object_perception-precise-amd64
65M doc-fuerte-topological_navigation
142M    doc-fuerte-topological_navigation-precise-amd64
61M doc-fuerte-trajectory_filters-precise-amd64
756K    doc-fuerte-tu-darmstadt-ros-pkg-precise-amd64
70M doc-fuerte-turtlebot
211M    doc-fuerte-turtlebot_apps-precise-amd64
149M    doc-fuerte-turtlebot_arm
144M    doc-fuerte-turtlebot-precise-amd64
72M doc-fuerte-turtlebot_simulator
62M doc-fuerte-turtlebot_simulator-precise-amd64
117M    doc-fuerte-turtlebot_viz-precise-amd64
37M doc-fuerte-universal_robot-precise-amd64
24M doc-fuerte-vcstools
24M doc-fuerte-vcstools-precise-amd64
11M doc-fuerte-velodyne-precise-amd64
32M doc-fuerte-velodyne_utils-precise-amd64
55M doc-fuerte-vision_opencv
53M doc-fuerte-vision_opencv-precise-amd64
78M doc-fuerte-vision_visp
127M    doc-fuerte-vision_visp-precise-amd64
40M doc-fuerte-viso2-precise-amd64
187M    doc-fuerte-visualization_engine-precise-amd64
117M    doc-fuerte-visualization_experimental
202M    doc-fuerte-visualization_experimental-precise-amd64
352M    doc-fuerte-visualization-precise-amd64
138M    doc-fuerte-vslam-precise-amd64
32M doc-fuerte-warehousewg-precise-amd64
160M    doc-fuerte-web_interface-precise-amd64
24M doc-fuerte-wg_common
27M doc-fuerte-wg_common-precise-amd64
52M doc-fuerte-wge100_driver
4.0K    doc-fuerte-wge100_driver-precise-amd64
452M    doc-fuerte-wg_hardware_test
1012M   doc-fuerte-wg_hardware_test-precise-amd64
57M doc-fuerte-wg_pr2_apps-precise-amd64
323M    doc-fuerte-wg_robots_gazebo-precise-amd64
18M doc-fuerte-wifi_drivers-precise-amd64
110M    doc-fuerte-worldmodel
84M doc-fuerte-worldmodel-precise-amd64
42M doc-fuerte-xacro-precise-amd64
34M doc-fuerte-zeroconf_avahi_suite
7.5M    doc-fuerte-zeroconf_implementations-precise-amd64
34M doc-groovy-ackermann_msgs-precise-amd64
93M doc-groovy-actionlib
56M doc-groovy-arbotix-precise-amd64
40M doc-groovy-arm_navigation_experimental-precise-amd64
85M doc-groovy-arm_navigation-precise-amd64
258M    doc-groovy-art_vehicle-precise-amd64
22M doc-groovy-bfl-precise-amd64
37M doc-groovy-bosch_3rdparty
45M doc-groovy-brown_perception
24M doc-groovy-brown_perception-precise-amd64
43M doc-groovy-camera1394-precise-amd64
23M doc-groovy-camera_info_manager_py
77M doc-groovy-camera_pose
25M doc-groovy-catkin
51M doc-groovy-catkin-precise-amd64
57M doc-groovy-clearpath_common-precise-amd64
143M    doc-groovy-clearpath_husky-precise-amd64
33M doc-groovy-clearpath_kingfisher
26M doc-groovy-clearpath_turtlebot-precise-amd64
31M doc-groovy-cob_calibration_data
24M doc-groovy-cob_command_tools
66M doc-groovy-cob_command_tools-precise-amd64
768K    doc-groovy-cob_environments
31M doc-groovy-cob_extern-precise-amd64
73M doc-groovy-cob_navigation
163M    doc-groovy-cob_navigation-precise-amd64
101M    doc-groovy-cob_robots
768K    doc-groovy-cob_simulation
63M doc-groovy-cob_simulation-precise-amd64
23M doc-groovy-common_msgs-precise-amd64
21M doc-groovy-common_tutorials-precise-amd64
25M doc-groovy-control-precise-amd64
73M doc-groovy-diagnostics
22M doc-groovy-diagnostics_monitors
46M doc-groovy-diagnostics_monitors-precise-amd64
68M doc-groovy-diagnostics-precise-amd64
60M doc-groovy-driver_common
43M doc-groovy-driver_common-precise-amd64
41M doc-groovy-dynamixel_motor-precise-amd64
222M    doc-groovy-ecl_core-precise-amd64
58M doc-groovy-ecl_lite
30M doc-groovy-ecl_manipulation
59M doc-groovy-ecl_navigation
42M doc-groovy-ecl_tools-precise-amd64
102M    doc-groovy-erratic_robot-precise-amd64
122M    doc-groovy-ethzasl
38M doc-groovy-ethzasl_aseba-precise-amd64
223M    doc-groovy-ethzasl-precise-amd64
28M doc-groovy-ethzasl_xsens_driver
27M doc-groovy-ethzasl_xsens_driver-precise-amd64
33M doc-groovy-executive_smach
21M doc-groovy-executive_smach_visualization
25M doc-groovy-executive_smach_visualization-precise-amd64
21M doc-groovy-executive_teer
21M doc-groovy-executive_teer-precise-amd64
39M doc-groovy-filters
21M doc-groovy-flirtlib_features
21M doc-groovy-flirtlib_features-precise-amd64
31M doc-groovy-gencpp
23M doc-groovy-gencpp-precise-amd64
24M doc-groovy-genlisp-precise-amd64
27M doc-groovy-genpy
113M    doc-groovy-geometry_experimental
99M doc-groovy-geometry_experimental-precise-amd64
22M doc-groovy-geometry_tutorials
19M doc-groovy-geometry_tutorials-precise-amd64
34M doc-groovy-geometry_visualization
29M doc-groovy-gps_drivers
20M doc-groovy-gps_drivers-precise-amd64
34M doc-groovy-gps_umd-precise-amd64
23M doc-groovy-humanoid_msgs-precise-amd64
4.1M    doc-groovy-ias_common-precise-amd64
2.2M    doc-groovy-image_common-precise-amd64
2.2M    doc-groovy-image_pipeline
3.8M    doc-groovy-image_pipeline-precise-amd64
40M doc-groovy-image_transport_plugins
34M doc-groovy-imu_drivers-precise-amd64
2.2M    doc-groovy-interactive_markers
2.2M    doc-groovy-interactive_markers-precise-amd64
5.4M    doc-groovy-joystick_drivers-precise-amd64
691M    doc-groovy-knowrob
314M    doc-groovy-knowrob-precise-amd64
3.9M    doc-groovy-kobuki_desktop
62M doc-groovy-kobuki-precise-amd64
49M doc-groovy-laser_assembler-precise-amd64
36M doc-groovy-laser_drivers-precise-amd64
34M doc-groovy-laser_filters
27M doc-groovy-map_manager_app-precise-amd64
54M doc-groovy-map_store
44M doc-groovy-map_store-precise-amd64
26M doc-groovy-megatree_pcl
756K    doc-groovy-multimaster_experimental-precise-amd64
33M doc-groovy-multimaster_fkie-precise-amd64
204M    doc-groovy-navigation
33M doc-groovy-navigation_experimental-precise-amd64
383M    doc-groovy-navigation-precise-amd64
21M doc-groovy-netft-precise-amd64
31M doc-groovy-nmea_gps_driver
24M doc-groovy-nmea_gps_driver-precise-amd64
216M    doc-groovy-object_manipulation-precise-amd64
37M doc-groovy-occupancy_grid_utils
47M doc-groovy-occupancy_grid_utils-precise-amd64
40M doc-groovy-octomap_mapping
24M doc-groovy-octomap_mapping-precise-amd64
32M doc-groovy-openni_camera-precise-amd64
25M doc-groovy-openni_launch-precise-amd64
26M doc-groovy-openni_tracker-precise-amd64
82M doc-groovy-orocos_kinematics_dynamics-precise-amd64
24M doc-groovy-pluginlib-precise-amd64
94M doc-groovy-pr2_apps
19M doc-groovy-pr2_apps-precise-amd64
321M    doc-groovy-pr2_arm_navigation
121M    doc-groovy-pr2_arm_navigation-precise-amd64
27M doc-groovy-pr2_calibration
98M doc-groovy-pr2_calibration-precise-amd64
95M doc-groovy-pr2_common_actions-precise-amd64
238M    doc-groovy-pr2_common-precise-amd64
22M doc-groovy-pr2_controllers-precise-amd64
20M doc-groovy-pr2_ethercat_drivers-precise-amd64
42M doc-groovy-pr2_kinematics
78M doc-groovy-pr2_kinematics-precise-amd64
29M doc-groovy-pr2_make_a_map_app
34M doc-groovy-pr2_make_a_map_app-precise-amd64
39M doc-groovy-pr2_mannequin_mode_app
19M doc-groovy-pr2_mechanism-precise-amd64
45M doc-groovy-pr2_object_manipulation
25M doc-groovy-pr2_pan_tilt-precise-amd64
340M    doc-groovy-pr2_plugs
57M doc-groovy-pr2_power_drivers
52M doc-groovy-pr2_power_drivers-precise-amd64
34M doc-groovy-pr2_props_app-precise-amd64
24M doc-groovy-pr2_ps3_joystick_app
30M doc-groovy-pr2_ps3_joystick_app-precise-amd64
100M    doc-groovy-pr2_self_test
39M doc-groovy-pr2_teleop_app
29M doc-groovy-pr2_teleop_app-precise-amd64
113M    doc-groovy-pr2_web_apps-precise-amd64
57M doc-groovy-prosilica_driver
28M doc-groovy-reconfigure_gui
39M doc-groovy-reconfigure_gui-precise-amd64
34M doc-groovy-remote_lab-precise-amd64
50M doc-groovy-riq_hand-precise-amd64
15M doc-groovy-robot_model_tutorials-precise-amd64
31M doc-groovy-robot_model_visualization
25M doc-groovy-robot_model_visualization-precise-amd64
32M doc-groovy-rocon_comms-precise-amd64
236M    doc-groovy-ros_comm
691M    doc-groovy-ros_comm-precise-amd64
33M doc-groovy-roscpp_core
18M doc-groovy-roscpp_core-precise-amd64
20M doc-groovy-rosdoc_lite-precise-amd64
25M doc-groovy-rosh_core-precise-amd64
35M doc-groovy-rosh_desktop_plugins-precise-amd64
36M doc-groovy-roshpit-precise-amd64
62M doc-groovy-rosh_robot_plugins
39M doc-groovy-rosh_robot_plugins-precise-amd64
23M doc-groovy-roslisp_support
33M doc-groovy-rospack-precise-amd64
110M    doc-groovy-rosserial-precise-amd64
23M doc-groovy-ros_tutorials
21M doc-groovy-ros_tutorials-precise-amd64
163M    doc-groovy-rviz-precise-amd64
64M doc-groovy-rx-precise-amd64
143M    doc-groovy-schunk_modular_robotics
274M    doc-groovy-schunk_modular_robotics-precise-amd64
53M doc-groovy-schunk_robots
72M doc-groovy-simple_arms
53M doc-groovy-simple_arms-precise-amd64
42M doc-groovy-slam_gmapping-precise-amd64
31M doc-groovy-sql_database-precise-amd64
36M doc-groovy-stage
28M doc-groovy-stage-precise-amd64
13M doc-groovy-std_msgs-precise-amd64
126M    doc-groovy-symbolic_planning
222M    doc-groovy-symbolic_planning-precise-amd64
78M doc-groovy-turtlebot_apps
68M doc-groovy-turtlebot_simulator
38M doc-groovy-turtlebot_simulator-precise-amd64
36M doc-groovy-turtlebot_viz-precise-amd64
31M doc-groovy-unique_identifier-precise-amd64
53M doc-groovy-universal_robot-precise-amd64
57M doc-groovy-velodyne
90M doc-groovy-velodyne-precise-amd64
30M doc-groovy-velodyne_utils
20M doc-groovy-velodyne_utils-precise-amd64
2.2M    doc-groovy-vision_opencv-precise-amd64
24M doc-groovy-vision_visp-precise-amd64
81M doc-groovy-visualization_common-precise-amd64
45M doc-groovy-warehousewg-precise-amd64
96M doc-groovy-web_interface-precise-amd64
24M doc-groovy-wg_common
21M doc-groovy-wg_common-precise-amd64
45M doc-groovy-wge100_driver
22M doc-groovy-wge100_driver-precise-amd64
21M doc-groovy-wg_pr2_apps
32M doc-groovy-wg_pr2_apps-precise-amd64
30M doc-groovy-wifi_drivers
50M doc-groovy-win_ros
25M doc-groovy-xacro-precise-amd64
31M doc-groovy-zeroconf_avahi_suite-precise-amd64
103M    doc-groovy-zeroconf_implementations-precise-amd6
...
tfoote commented 11 years ago

Note this is accumulating on the slaves.

eitanme commented 11 years ago

Should I be deleting things after a successful run or something? I'm fairly sure all the data is important... I guess we don't need to keep that stuff around for successful runs?

tfoote commented 11 years ago

I'm surprised about some of the sizes, a few of them exceeded a gigabyte, which seems overly large for generating documentation. Which in itself might warrent looking into.

As these should be gernerated files and they're going to need to be regenerated the next time around, probably on another machine it's probably not worth keeping them. If you have the source checkout and can not require a full recheckout that would be good not to delete.

On Fri, Nov 16, 2012 at 6:27 PM, eitanme notifications@github.com wrote:

Should I be deleting things after a successful run or something? I'm fairly sure all the data is important... I guess we don't need to keep that stuff around for successful runs?

— Reply to this email directly or view it on GitHubhttps://github.com/ros-infrastructure/rosdoc_lite/issues/7#issuecomment-10469567.

Tully Foote tfoote@willowgarage.com (650) 475-2827

eitanme commented 11 years ago

A couple things after digging around a bit:

I'm going to modify the doc script to remove the auto-generated files after they are successfully uploaded.

eitanme commented 11 years ago

This shouldn't be an issue now that the jobs remove their data... closing.

tfoote commented 11 years ago

I confirmed that the new jobs are cleaning up relatively well now. We just have a lot of jobs these days.