search

rueckstiess / mtools

A collection of scripts to set up MongoDB test environments and parse and visualize MongoDB log files.
Apache License 2.0
1.89k stars 403 forks source link

Project dependencies may have API risk issues #886

Closed PyDeps closed 2 years ago

PyDeps commented 2 years ago

Hi, In mtools, inappropriate dependency versioning constraints can cause risks.

Below are the dependencies and version constraints that the project is using

ordereddict>=1.1
python-dateutil>=2.8.2
numpy>=1.21.4
matplotlib>=3.5.0
pymongo>=3.12.2<5.0.0
psutil>=5.8.0

The version constraint == will introduce the risk of dependency conflicts because the scope of dependencies is too strict. The version constraint No Upper Bound and * will introduce the risk of the missing API Error because the latest version of the dependencies may remove some APIs.

After further analysis, in this project, The version constraint of dependency numpy can be changed to >=1.16.0rc1,<=1.18.5. The version constraint of dependency matplotlib can be changed to >=1.3.0,<=3.0.3. The version constraint of dependency psutil can be changed to >=1.2.0,<=5.9.1.

The above modification suggestions can reduce the dependency conflicts as much as possible, and introduce the latest version as much as possible without calling Error in the projects.

The invocation of the current project includes all the following methods.

The calling methods from the numpy 
c
char.isdigit
distutils.core.setup
The calling methods from the matplotlib 
matplotlib.patches.Polygon
matplotlib.dates.num2date
matplotlib.dates.AutoDateLocator
matplotlib.dates.date2num
matplotlib.dates.AutoDateFormatter
The calling methods from the psutil 
psutil.wait_procs
psutil.process_iter
The calling methods from the all methods 
self.coll_handle.count_documents
json.loads
artist.set_alpha
sys.path.insert
presplit
self._construct_sharded
io.BytesIO
doc.str.replace
platform.group.group
os.path.basename
matplotlib.dates.AutoDateFormatter
ProfileCollection
self.remove_overlays
mtools.mplotqueries.plottypes.base_type.BasePlotType.plot
set.add
tuple
datetime.datetime.fromtimestamp
DateTimeBoundaries.string2dt
self.stop
re.match.groups
line.find
axis.get_ylim
LogTuple
self.plot_instance.accept_line
self._get_tls_pymongo_options
dateutil.tz.tzutc.self.datetime.astimezone.strftime
line.lstrip.lstrip
self._start_on_ports
mc.admin.command
matplotlib.pyplot.annotate
bson.decode
self._strip_counters
argparse.ArgumentParser.add_argument
versions.append
OSError
collections.defaultdict.keys
version.self.matches.append
logfile.filehandle.seek
self.groups.keys
list.append
dt.replace.replace
self._construct_config
logline.rindex
self.ignored_arguments.get
legend_line.set_picker
timestamps.append
subparsers.add_parser.add_argument_group
property.rfind
matplotlib.pyplot.gca.set_yscale
artists.extend
self.logfile.start.strftime
subprocess.check_call
matplotlib.dates.date2num.append
matplotlib.pyplot.gca
self.doExport
op_or_cmd
self._datetime_nextpos.split_tokens.lower
MLaunchTool.run
os.path.exists
collection.database.con.create_indexes
self._find_curr_line.set_datetime_hint
source_files
float
list.key.self.groups.setdefault.append
nodes.intersection.pop
self.client.close
opened_closed
pymongo.MongoClient
out.decode.split
self.datetime.utcoffset
sorted.append
self.cursor.next
wiredtiger.wiredtiger_open.open_session
self._thread.startswith
f
split.time.strftime
io.BytesIO.readline
self.unknown_args.index
matplotlib.pyplot.subplot
self.loaded_args.get
dict.get
sum
axis.annotate
self._find_pattern
doc_or_str.decode.rstrip
sys.stderr.write
sys.exit
self._load_parameters
distutils.core.setup
eval
zip.append
dict
group_by
pymongo.Connection
matplotlib.dates.date2num
self._reformat_timestamp
self.get_tags_of_port.append
get_all_versions
self._match_datetime_pattern
counter_equiv.keys
datetime.datetime.now.replace
next.index
outf.write
matplotlib.pyplot.subplot.grid
vars
self.argparser.error
inspect.getmembers
self.mlogfilter.args.split
re.I.re.M.line.end_connid_pattern.search.group.isdigit
kwargs.setdefault
queue.Queue.get_nowait
matplotlib.pyplot.gca.add_patch
self._datetime_format.startswith
pickle.load
self._initiate_replset
kwargs.update
self.plot_total_conns
matplotlib.patches.Polygon
matplotlib.pyplot.savefig
logline.index
matplotlib.pyplot.switch_backend
matplotlib.pyplot.subplot.legend
SplitTuple
reversed
split_words.logs_by_word.append
ValueError
matplotlib.pyplot.gca.get_yscale
axis.axes.get_yaxis
self.plot_instances.extend
artist.get_alpha
join
artist.get_markeredgewidth
self.groups.move_items
self._changeMs
self.argparser.print_help
driver_info.get
section.run
x_rights.append
re.subn
line.decode.strip
datasets.append
self.next
argparse.ArgumentTypeError
a.get_height
numpy.percentile
logevent.line_str.lower
x.time.strftime
arg.startswith
dt_string.split.endswith
self.line_str.rfind
self.datetime.strftime.split
next
ImportError
axis.semilogy
time.time
self.datetime.astimezone
len
dateutil.tz.tzutc
a.set_height
self._parse_document
main
conn.open_session.commit_transaction
colors.append
MongoConnection
get_all_versions.index
self._reset
pymongo.Connection.server_info
logevent.datetime.strftime
ChunksTuple
dt_string.split.replace
self.start
self._arrayToString
self.load_overlays
pymongo.Connection.update_one
range
self.coll_handle.find
int
re.search.groups
datetime_filter.DateTimeFilter.__init__
stop_idx.search_str.strip
self.cluster_tags.extend
InputSourceAction
version.group.group
self.toggle_marker_edges
sub_line.index
all
self.log_levels.index
base_extras_requires.copy
driver_info.items
self.args.self.plot_types
self.set_markersizes
re.search.group
subprocess.Popen.communicate
get_all_versions.append
open.readlines
time.sleep
plot_inst.plot
accepted_arguments.append
re.finditer
pattern.log_code_lines.addMatch
MLogInfoTool.run
str
artist.set_visible
self._merge_logfiles
self._extract_operation_and_namespace
matplotlib.dates.AutoDateLocator
_decode_pattern_dict
f.read
re.sub
datetime.datetime.now.strftime
token.startswith
matplotlib.pyplot.subplots_adjust
collections.defaultdict.values
mongod_matches.intersection.intersection
self._find_curr_line
print_table
self._any_markersizes_to_increase
matplotlib.pyplot.subplot.set_xticklabels
self.mloginfo._datetime_to_epoch
self.message
abs
self.filehandle.readlines
print_docs.append
mtools.util.cmdlinetool.InputSourceAction
dateutil.parser.parse
self._any_markersizes_to_decrease
list.remove
setuptools.find_packages
newline_pos.buff.rfind
self.mloginfo.logfile.chunk_splits.reverse
wiredtiger.wiredtiger_open.close
self.increase_marker_size
self.coll_handle.find_one
split
check_mongo_server_output.decode
doc.str.replace.replace
newfile_meta.re.search.group
self.coll_handle.find.sort
self.legend.get_lines
matplotlib.pyplot.gcf
p.cmdline
conn.open_session.create
socket.gethostname
self.datetime.weekday
self.getMongoDVersion
line_str.rfind.line_str.replace
self.plot_instance.add_line
chr
pickle.dump
list.to_group.self.groups.setdefault.extend
others_label.self.groups.extend
replace.items
logfile.filehandle.tell
subprocess.Popen
distutils.sysconfig.get_config_vars
self.get_tagged
logfile.fast_forward
plot_inst.group
mtools.util.cmdlinetool.LogFileTool.__init__
mtools.util.logcodeline.LogCodeLine
arg.split
self.filters.append
self._construct_mongod
wiredtiger.wiredtiger_open
base_section.BaseSection.__init__
self.argparser.add_argument
map
char.isdigit
self.mlogfilter.args.filehandle.tell
buff.decode.rfind
self._doImport
self._add_user
export.items
self.keys
self._calculate_bounds
self.argparser.parse_args
axis.twinx
mtools.mplotqueries.plottypes.base_type.BasePlotType.plot.append
enumerate
mtools.util.OrderedDict
itertools.zip_longest
self.args.get
buf.readline.strip
token.split.replace
event.artist.get_markerfacecolor
self.cluster_tags.append
os.path.join.split
ns.startswith
regex.search.end
self._construct_mongos
self._restarts.append
self.cluster_tree.setdefault
logpath.replace
matplotlib.pyplot.gcf.canvas.draw
group.replace
thread.start
event.artist.get_visible
re.match.group
self._init_opacities
artist.set_markeredgewidth
re.sub.rfind
numpy.linspace
self.mloginfo.update_progress
re.findall
self.logfile_generator
MLogInfoTool
os.remove
self.log2code
self._shards.append
self.datetime.strftime
self.print_shortcuts
self.plot
self.init
data.items
self._variable_parts
self._construct_cmdlines
self.client.count_documents
pymongo.Connection.command
artist.get_markersize
self._store_parameters
self.color_map
first_token.split
result.groupdict
difflib.SequenceMatcher
self.plot_instances.append
pos.pos.line.split
vars.split
mtools.mplotqueries.plottypes.base_type.BasePlotType.__init__
self.end.replace.weekday
tests.items
numpy.min
matplotlib.pyplot.figure
self.update_progress
self.toggle_artist
argparse.FileType.__call__
self.argparser.add_argument_group
self.decrease_opacity
os.path.join
functools.lru_cache
row.items.items
argparse.ArgumentParser.parse_args
mtools.util.cmdlinetool.LogFileTool.run
axis.barh
doc_or_str.decode.decode
os.path.join.replace
pymongo.Connection.aggregate
self._any_opacities_to_increase
idx.app_metadata_re.search.group
matplotlib.pyplot.gca.set_ylim
chunk.time.strftime
datetime.timedelta
counter_equiv.get
regex.search.start
threads.append
MLogFilterTool
export.app_metadata_re.search.group
matplotlib.pyplot.subplot.set_title
arguments.startswith
l.logs_versions.append
sorted
pymongo.Connection.find
p.cmdline.index
dt_string.split.startswith
logevent.to_json
custom_parse_array
axis.set_zorder
bson.SON
self.groups.setdefault
dateutil.utils.default_tzinfo
axis.hist
mtools.util.grouping.Grouping.sort_by_size
self._rs_state.append
self.argparser.print_usage
tokens.strip.isdigit
self.start.replace
dict.items
pymongo.Connection.find_one
queue.Queue.put_nowait
axis.autoscale_view
matplotlib.pyplot.gcf.canvas.mpl_connect
MLogFilterTool.run
self.mloginfo.argparser_sectiongroup.add_argument
self._chunks_moved_from.append
matplotlib.dates.num2date.strftime
event.artist._mt_left.num2date.strftime
mtools.util.logevent.LogEvent
timestamp.strftime.strftime
self._get_tls_server_args
os.removedirs
s.re.sub.strip
self._wait_for_primary
group.strftime
artist.set_markersize
self.argparser.parse_known_args
name.self.config_docs.append
line.split.endswith
bson.decode.get
self._check_for_restart
self.artists.extend
self._pad_event
matplotlib.pyplot.subplot.set_ylim
command.lower
pymongo.MongoClient.__init__
SystemExit
bson.decode_file_iter
self._extract_counters
self._formatNumbers.rindex
numpy.max
self._find_variable
self.get_tagged.intersection
datetime.datetime.now
self.doImport
mtools.util.print_table.print_table
result.group
result.append
self.to_dict
open
self._get_ports_from_args
obj.isoformat
mask_list.append
self.plot_types.keys
sys.stderr.flush
MTransferTool
matplotlib.pyplot.subplot.set_ylabel
check_mongo_server_output
datetime.datetime.now.tzname
os.path.expanduser
subparsers.add_parser.add_argument
self.cluster_tree.setdefault.append
buff.decode.decode
collections.namedtuple
namespace.split
MTransferTool.run
mtools.util.log2code.Log2CodeConverter
platform.python_compiler.lower
datetime.datetime
self._strip_datetime.strip
conn.open_session.drop
type
self._construct_single
day.isdigit
os.path.dirname
set.update
instance.instance.mc.update_one
property.find
self.increase_opacity
bool
plot_type.clicked
bson.son.SON
key.encode.encode
line.decode.decode
error_log_line.errorline.SequenceMatcher.ratio
round
platform.python_compiler
matplotlib.__version__.split
LogFileTool.run
str.encode
self.shard_connection_str.append
self._datetime_nextpos.self.split_tokens.isdigit
dtbound
self._find_sharding_info
self._outputLine
match.group.re.sub.strip.group
len.endswith
axis.xaxis.set_major_locator
le.line_str.re.search.group
artist.get_visible
self.client
mtools.util.grouping.Grouping.items
self.filehandle.seek
_decode_pattern_list
axis.axvline
self._chunks_moved_to.append
con.admin.command
min
self._formatNumbers
self._print_chunk_migrations
result.group.strip
operator.itemgetter
self._init_markersizes
mtools.util.cmdlinetool.BaseCmdLineTool.__init__
self._create_paths
self._datetime.replace
self.kill
hasattr
self._read_key_file
self._get_shard_names
pymongo.MongoClient.update_one
mtools.util.grouping.Grouping.add
mtools.util.grouping.Grouping
init_parser.add_mutually_exclusive_group.add_argument
self._line_str.split
matplotlib.pyplot.subplot.get_xticks
session.open_cursor.prev
self.string2dt
re.sub.split
session.open_cursor.get_key
sys.exc_info
any
section.name.upper
self.argparser.add_subparsers.add_parser
port.str.self.cluster_tags.append
self.groups.get
self.datetime.isoformat
match.group.re.sub.strip
re.match.start
self.parse_logevents
self.list_overlays
mtools.util.cmdlinetool.BaseCmdLineTool.run
line_str.strip.append
re.search
init_parser.add_argument_group.add_argument
database.con.command
queue.Queue
getattr
self.groups.sort_by_size
bson.codec_options.CodecOptions
mo.start.strip
psutil.wait_procs
self.client.command
matplotlib.pyplot.subplot.get_legend_handles_labels
self.weekdays.index
Grouping
warnings.warn
self._iterate_lines
pos.pos.line.split.strip
subparsers.add_parser.error
axis.twinx.set_ylabel
distutils.version.LooseVersion
self.filehandle.read
self.end.replace
re.I.re.M.line.end_connid_pattern.search.group
dv_formatted.driver_info.append
uuid.uuid4
ScatterPlotType.__init__
self.durlines.itemgetter.map.index
format
current_version.rpartition.rpartition
self._extract_level
self.plot_group
self.discover
roles.append
default.replace.replace
format.startswith
logevent._reformat_timestamp
argparse.ArgumentParser
possible_port.self.startup_info.split.index
self.set_opacities
self.l2cc
re.search.start
install_requires.append
line.decode.split
shell2json
subparsers.add_parser.add_mutually_exclusive_group
self._chunk_splits.append
mtools.util.logfile.LogFile
bson.decode.items
os.makedirs
mtools.util.hci.DateTimeBoundaries
thread.join
self.datetime_nextpos.t.split_tokens.replace
self.filehandle.readline
matplotlib.patches.Polygon.remove
chunks.reverse
self.durlines.append
join.format
sys.argv.startswith
axis.xaxis.set_major_formatter
axis.plot_date
os.path.abspath
artists.append
counters.extend
matplotlib.pyplot.autoscale
self.restart
import_l2c_db
iter
self.save_overlay
RuntimeError
matches.append
axis.twinx.get_zorder
json.dump
property.rstrip
dtb
self.is_running
psutil.process_iter
command_str.split.endswith
self.add
self.group
tag.split
doc.processes.cmdline
self._start.replace
pymongo.MongoClient.drop
self.get_tags_of_port
self._get_processes.keys
json.dumps
zip
conn.open_session.open_cursor
mtools.util.pattern.json2pattern
token.split
table_rows.append
threading.Thread
json2pattern
buf.readline.strip.decode
property
matplotlib.pyplot.subplot.set_xlim
DateTimeBoundaries
self.split_tokens.index
f.accept
event.artist._mt_right.num2date.strftime
re.compile.search
self.filehandle.tell
self.wait_for
exec
self._get_ssl_server_args
axis.plot
logevent.line_str.self.field.re.search.group
self.datetime.microsecond.int.str.zfill
self.versions.add
print
line.lstrip.startswith
bson.encode
base_filter.BaseFilter.__init__
subprocess.check_output
x.upper
sys.stdin.isatty
self.groups.items
mtools.util.profile_collection.ProfileCollection
glob.glob
os.walk
Grouping.regroup
i.self.cluster_tree.append
set
BaseCmdLineTool.__init__
out.split
database.mc.command
os.system
sub_line.rindex
coverage.append
c
self._strip_datetime
collections.defaultdict
json.JSONEncoder.default
self.decrease_marker_size
config_string.append
distutils.spawn.find_executable
self._get_last_error_log
line.decode.rstrip
int.group
self.groups.values
dict.keys
Grouping.sort_by_size
list
coverage.index
args.get
self._msToString
self.argparser.add_subparsers
p.send_signal
doc.keys
a.get_visible
self.dbpath.os.path.join.open.read
property.endswith
divmod
LogFileTool
self.plot_type_str.upper
cls.states.index
re.escape
regex.search
axis.axes.get_yaxis.set_visible
NotImplementedError
MPlotQueriesTool
switch_version
self._get_log_path
value.split
self._log2code
self.args.self.getattr
self._get_processes
json.load
math.ceil
axis.patch.set_visible
conn.open_session.begin_transaction
extensions.append
axis.set_ylim
current_version.rpartition.rindex
p.name
self._get_ssl_pymongo_options
signal.signal
line.lstrip.split
self._end.replace
nodes.intersection.intersection
f.skipRemaining
trigger.len.trigger.statement.find.statement.strip
running.self.cluster_tags.append
MPlotQueriesTool.run
self.cluster_tree.append
pos.pos.line.split.split
self._print_chunk_statistics
max
self._construct_replset
arguments.split
self._datetime_to_epoch
self._any_opacities_to_decrease
re.match
self.argparser.exit
self._filter_valid_arguments
split_tokens.index
re.split
possible_port.self.startup_info.split
matplotlib.pyplot.subplot.set_xlabel
f.setup
isinstance
f.readlines
matplotlib.dates.num2date
re.sub.find
trigger.strip.strip
shard.self.cluster_tags.extend
bson.min_key.MinKey
MLaunchTool
self.logfile.end.strftime
shutdown_host
re.compile
matplotlib.pyplot.show
open.close
extract_logs

@developer Could please help me check this issue? May I pull a request to fix it? Thank you very much.

stennie commented 2 years ago

Hi @PyDeps,

Thank you for the suggestions. However, I believe your analysis is flawed as dependencies also relate to the versions of Python supported and tested by packages, as well as important bug fixes that may be included without changing the package API.

Taking psutil as an example, psutil 1.2.0 was released in Nov 2013 and there are years of bug fixes including a few issues that have affected this project. You have also chosen a few version ranges that do not include the current dependency versions tested.

The current constraints are for the oldest versions supported and tested. I can add upper bounds for semantic versioning, but this has not been a historic source of bugs for the dependencies used in this project.

Regards, Stennie