BillSum dataset finetuning

[sshleifer]

(from distilbart issue).

Hey @sshleifer , thanks for the distilled BART version I was able to fine tune it with the same script on BillSum dataset as T5 but the numbers are way different between the two. I just wanted to understand if I might be doing something wrong with regards to fine tuning distilBART, does it require student training everytime? Reference numbers on BillSum Dataset:

T5-base: avg_train_loss = tensor(1.5333, device='cuda:0') avg_val_loss = tensor(1.4528, device='cuda:0') epoch = 1 loss = tensor(1.6734, device='cuda:0') rouge1 = 0.49188267841912325 rouge2 = 0.26436589848185027 rougeL = 0.3591894400892483 train_loss = tensor(1.6734, device='cuda:0') val_loss = tensor(1.4528, device='cuda:0')

dBART-cnn-12-6: avg_train_loss = tensor(1.3013, device='cuda:0') avg_val_loss = tensor(1.4013, device='cuda:0') epoch = 1 loss = tensor(1.4901, device='cuda:0') rouge1 = 0.3681518923769047 rouge2 = 0.15683286277623087 rougeL = 0.23453727441540043 train_loss = tensor(1.4901, device='cuda:0') val_loss = tensor(1.4013, device='cuda:0')

PS. I am using a modified version of the older finetune.py so it doesn't have Rouge for validation epochs.

Thanks

[sshleifer]

Interesting! I wonder what you get starting from bart-large. If you use the new finetune.py with --logger wandb I could more easily see your logs and suggest hyperparameters.

[amanpreet692]

Thanks! and Sure, let me do that and share the results soon!

[amanpreet692]

Took me a while to get to this :/ the numbers look as expected when fine-tuned with the latest finetune.py! :) Must have been a bug in my modified script, updated that too and confirmed the results: dBART: "val_avg_loss": 1.3925694227218628, "val_avg_loss": 1.3925694227218628, "val_avg_rouge1": 0.576209431885421, "val_avg_rouge2": 0.34462863342832595, "val_avg_rougeL": 0.3999874418853423, "val_avg_gen_time": 6.152730942964554, "val_avg_summ_len": 132.768

T5: "val_avg_loss": 1.6064366102218628, "val_avg_rouge1": 0.5291717003599621, "val_avg_rouge2": 0.3049920551697407, "val_avg_rougeL": 0.37437278542325914, "val_avg_gen_time": 2.408391712649965, "val_avg_summ_len": 171.18881856540085

Just a quick query though if you could help me out, for a lot of the predictions in the resulting test_generation file, I am getting summaries starting with lower case, and then generating a coherent summary from there on. I looked at my dataset and the code and fine-tuned the models again after some pre-processing but still the same result. Any ideas?? This was for both T5 and dBart btw.

Thanks!

[sshleifer]

Can I see an example of an input, output pair that demonstrates the issue you are facing?

[amanpreet692]

Here you go...apologies I should have shared the examples in the previous post itself :) Example 1: Src : This Act may be cited as the "Military Call-Up Relief Act". (a) Waiver For Certain Distributions. (1) In general. Section 72(t)(2) of the Internal Revenue Code of 1986 is amended by adding at the end the following: (G) Distributions to individuals performing national emergency active duty. Any distribution to an individual who, at the time of the distribution, is a member of a reserve component called or ordered to active duty pursuant to a provision of law referred to in section 101(a)(B) of title 10, United States Code, during the period of the national emergency declared by the President on September 14, 2001. (2) Waiver of underpayment penalty. Section 6654(e)(3) of such Code is amended by adding at the end the following: (C) Certain early withdrawals from retirement plans. No addition to tax shall be imposed under subsection (a) with respect to any underpayment to the extent such underpayment was created or increased by any distribution described in section 72(t)(2)(G). (3) Effective date. The amendments made by this subsection shall apply to distributions made to an individual after September 13, 2001. (b) Catch-up Contributions Allowed. (1) Individual retirement accounts. Section 219(b)(5) of the Internal Revenue Code of 1986 is amended by adding at the end the following: (D) Catch-up contributions for certain distributions. In the case of an individual who has received a distribution described in section 72(t)(2)(G), the deductible amount for any taxable year shall be increased by an amount equal to (i) the aggregate amount of such distributions made with respect to such individual, over " the aggregate amount of such distributions previously taken into account under this subparagraph or section 414(w). (2) Roth iras. Section 408A(c) of such Code is amended by redesignating paragraph (7) as paragraph (8) and by inserting after paragraph (6) the following: (7) Catch-up contributions for certain distributions. Any contribution described in section 219(b)(5)(D) shall not be taken into account for purposes of paragraph (2). (3) Employer plans. Section 414 of such Code is amended by adding at the end the following: (w) Catch-up contributions for certain distributions. (1) In general. An applicable employer plan shall not be treated as failing to meet any requirement of this title solely because the plan permits an applicable participant to make additional elective deferrals in any plan year. (2) Limitation on amount of additional deferrals. (A) In general. A plan shall not permit additional elective deferrals under paragraph (1) for any year in an amount greater than the lesser of (i) the applicable dollar amount, or " the excess of (I) the participant's compensation (as defined in section 415(c) for the year, over " any other elective deferrals of the participant for such year which are made without regard to this subsection. (B) Applicable dollar amount. For purposes of this paragraph, the applicable dollar amount with respect to a participant shall be an amount equal to (i) the aggregate amount of distributions described in section 72(t)(2)(G) made with respect to such participant, over " the aggregate amount of such distributions previously taken into account under this subsection or section 219(b)(5)(B). (3) Treatment of contributions. Rules similar to the rules of paragraphs (3) and (4) of subsection (v) shall apply with respect to contributions made under this subsection. (4) Definitions. For purposes of this subsection, the terms 'applicable employer plan' and 'elective deferral' have the same meanings given such terms in subsection (v)(6). (4) Conforming amendment. Section 414(v)(2)(A) of such Code is amended by inserting (other than deferrals under subsection " after "deferrals". (5) Effective date. The amendments made by this subsection shall apply to contributions in taxable years ending after December 31, 2001.

Ground Truth: Amend the internal revenue code of 1986 to provide a waiver of the early withdrawal penalty for distributions from qualified retirement plans to individuals called to active duty during the national emergency declared by the president on september 14, 2001, and for other purposes. Military Call-up Relief Act - Amends the Internal Revenue Code to waive the ten percent early withdrawal penalty for distributions from qualified retirement plans to individuals called to active duty during the national emergency declared by the President on September 14, 2001.

Prediction: military call-up relief act - Amends the Internal Revenue Code to allow a tax-exempt distribution to an individual who is a member of a reserve component called or ordered to active duty during the period of the national emergency declared by the President on September 14, 2001. Requires an employer plan to not be treated as failing for purposes of this Act. Provides for a catch-up contribution for certain distributions.

Example 2: Src: This Act may be cited as the "National Climate Service Act of 2009". The Congress finds the following: (1) Weather, climate change, and climate variability affect public safety, environmental services and security, human health, agriculture, energy use, water resources, and other factors vital to national security and human welfare. (2) Climate forecasts create opportunities for society to prepare, potentially reducing the costs of climate-related events, as well as serving national needs related to enhancing economic growth, managing risk, protecting life and property, and promoting environmental stewardship. (3) Information on predicted climate and climate impacts is not being fully disseminated or used well, despite the increasing predictability of climate. (4) The United States lacks adequate research, infrastructure, and coordinated outreach and communication mechanisms to meet national climate monitoring, prediction, and decision support needs for adapting to and mitigating the impacts of climate change and climate variability. (5) Increasing societal resilience to climate impacts requires understanding climate trends and variations as well as possible, understanding the impacts of climate on human and nonhuman systems, providing decision-relevant tools based on that information, and increasing society's capacity to act on that information. It is the purpose of this Act to establish a National Climate Service that will assist the Nation and the world in understanding, anticipating, and responding to climate, climate change, and climate variability and their impacts and implications. The Service shall inform the public through the sustained production and delivery of authoritative, timely, useful information about impacts on local, State, regional, tribal, national, and global scales. The Service shall be user-centric, by ensuring that the information is accessible, consistent with users' ability to respond, and based on user needs and limitations. The Service shall provide such usable information through a sustained network of observations, modeling, and research activities. (a) Establishment. (1) In general. The Secretary of Commerce shall establish within the Climate Program Office of the National Oceanic and Atmospheric Administration a National Climate Service not later than one year after the date of enactment of this Act. The Service shall include a national center and a network of regional and local facilities for operational climate observation, modeling, and research. (2) General purpose. The Service shall inform the public through the sustained production and delivery of authoritative, timely, useful information about impacts on local, State, regional, tribal, national, and global scales. (3) Specific services. The Service, at minimum, shall (A) serve as a clearinghouse and technical access point to stakeholders for regionally and nationally relevant information on climate, climate impacts, and adaptation, developing comprehensive databases of information relevant to specific regional and national stakeholder needs. (B) provide education on climate impacts, vulnerabilities, and application of climate information in decisionmaking. (C) design decision-support tools that facilitate use of climate information in stakeholders' near-term operations and long-term planning (D) facilitate user access to climate and climate impacts experts for technical assistance in use of climate information and to inform the climate forecast community of their information needs. (E) provide researcher, modeler, and observations experts access to users to help guide direction of research, modeling, and observation activities. And (F) propose and evaluate adaptation strategies for climate variability and change. (4) Specific functions. The Service, at minimum, shall (A) integrate global, national, and regional observations to produce information and assessments of use to stakeholders and researchers, (B) develop climate models for decision support. (C) perform basic and applied research on climate dynamics and impacts relevant to stakeholder interests. (D) create and maintain an operational delivery system and facilitate transition of new climate applications products to Service member agencies. (E) establish an atmospheric monitoring and verification program utilizing aircraft, satellite, ground sensors, ocean and coastal observing systems, and modeling capabilities to monitor, measure, and verify greenhouse gas concentrations and emissions throughout the global oceans and atmosphere. (F) develop and maintain a dialog among research teams, Federal agencies, and stakeholders for developing information relevant for planning and decisionmaking. (G) identify climate-related vulnerabilities and build national capacity to increase resilience. (H) articulate regional and national climate issues and concerns in regional and national policy arenas and facilitate regional-national communications on Service needs and performance. And (I) outreach to stakeholder groups. (b) Action Plan. Within 1 year after the date of enactment of this Act, the Secretary of Commerce shall submit to the Committee on Commerce, Science, and Transportation of the Senate and the Committee on Science and Technology of the House of Representatives a plan of action for the National Climate Service. The plan, at a minimum, shall (1) provide for the interpretation and communication of climate data, conditions, predictions, projections, and risks on an ongoing basis to decision and policy makers at the local, regional, and national levels. (2) design, deploy, and operate a national climate observing system that closes gaps in existing coverage. (3) support infrastructure and ability to archive and ensure the quality of climate data, and make federally funded model simulations and other relevant climate information available from Global Change Research Program activities and other sources. (4) include a program for long-term stewardship, quality control, development of relevant climate products, and efficient access to all relevant climate data, products, and model simulations. (5) establish a national coordinated modeling strategy, including a national climate modeling center to provide a dedicated capability for modeling and forecasting scenarios, and a regular schedule of projections on long-term and short- term time horizons over a range of scales, including regional scales. (6) improve integrated modeling, assessment, and predictive capabilities needed to document and forecast climate changes and impacts, and to guide national, regional, and local planning and decisionmaking. (7) provide a system of regular consultation and coordination with Federal agencies, States, tribes, nongovernmental organizations, the private sector, and the academic community to ensure (A) that the information requirements of these groups are well incorporated. And (B) timely and full sharing, dissemination and use of climate information and services in risk preparedness, planning, decisionmaking, and early warning and natural resources management, both domestically and internationally. (8) develop standards, evaluation criteria, and performance objectives to ensure that the Service meets the evolving information needs of the public, policy makers, and decisionmakers in the face of a changing climate, (9) develop funding estimates to implement the plan. And support competitive research programs that will improve elements of the Service described in this Act through the Climate Program Office within the Service headquarter function. (c) Director. The Administrator shall appoint a Director of the Service, who shall oversee all processes associated with managing the organization and executing the functions and actions described in this Act. (d) National Climate Service Advisory Council. The Administrator shall, in consultation with the Chairmen and ranking minority members of the Committee on Commerce, Science, and Transportation of the Senate and the Committee on Science and Technology of the House of Representatives, and the National Academy of Sciences, appoint the membership of a National Climate Service Advisory Council, with members serving 4-year terms, that shall include a diverse membership from appropriate Federal, State, and local government, universities, and nongovernment and private sectors who use climate information and cover a range of sectors, such as water, drought, fisheries, coasts, agriculture, health, natural resources, transportation, and insurance. The Council shall advise the Director of the Service of key priorities in climate-related issues that require the attention of the Service. The Council shall be responsible for promoting coordination across regional, national, and international concerns and the assessment of evolving information needs. Functions vested in any Federal officer or agency by this Act or under the program established under this Act may be exercised through the facilities and personnel of the agency involved or, to the extent provided or approved in advance in appropriation Acts, by other persons or entities under contracts or grant arrangements entered into by such officer or agency. The Secretary of Commerce shall prepare and submit to the President and the Congress, not later than March 31 of each year, a report on the activities conducted pursuant to this Act during the preceding fiscal year, including (1) a summary of the achievements of the National Climate Service during the previous fiscal year. And (2) an analysis of the progress made toward achieving the goals and objectives of the Service. (1) Administrator. The term "Administrator" means the Administrator of the National Oceanic and Atmospheric Administration. (2) Advisory council. The term "Advisory Council" refers to the National Climate Service Advisor Council. (3) Climate change. The term "climate change" means any change in climate over time, whether due to natural variability or as a result of human activity. (4) Director. The term "Director" means the director of the National Oceanic and Atmospheric Administration's National Climate Service. (5) Secretary. The term "Secretary" means the Secretary of Commerce. (6) Service. The term "Service" means the National Oceanic and Atmospheric Administration's National Climate Service. There are authorized to be appropriated to the Secretary to carry out this Act (1) $300,000,000 for fiscal year 2011. (2) $350,000,000 for fiscal year 2012. (3) $400,000,000 for fiscal year 2013. (4) $450,000,000 for fiscal year 2014. (5) $500,000,000 for fiscal year 2015.

Ground Truth: Provide for the establishment of a national climate service, and for other purposes. National Climate Service Act of 2009 - Requires the Secretary of Commerce to establish within the Climate Program Office of the National Oceanic and Atmospheric Administration a National Climate Service that includes a national center and a network of regional and local facilities for operational climate observation, modeling, and research. Requires the Service to: (1) inform the public about climate impacts. (2) serve as a clearinghouse and technical access point to stakeholders for information on climate, climate impacts, and adaptation, and relevant comprehensive databases of information. (3) provide education on climate impacts, vulnerabilities, and application of climate information in decisionmaking. (4) design decision-support tools that facilitate use of climate information in stakeholders' near-term operations and long-term planning. (5) facilitate user access to climate experts for technical assistance in the use of climate information and to inform the climate forecast community of their information needs. (6) provide researcher, modeler, and observations experts access to users to help guide direction of their activities. And (7) propose and evaluate adaptation strategies for climate variability and change. Sets forth the Service's functions, including establishing an atmospheric monitoring and verification program utilizing aircraft, satellite, ground sensors, ocean and coastal observing systems, and modeling capabilities to monitor, measure, and verify greenhouse gas concentrations and emissions throughout the oceans and atmosphere. Requires the Secretary to report to specified congressional committees on a plan of action for the Service. Requires the Administrator of NOAA to appoint a Director of the Service. Requires the Director to appoint members of a National Climate Service Advisory Council to promote coordination across regional, national, and international concerns and assess information needs.

Prediction: the Secretary of Commerce shall establish within the Climate Program Office of the National Oceanic and Atmospheric Administration a National Climate Service that will assist the Nation and the world in understanding, anticipating, and responding to climate, climate change, and climate variability and their impacts and implications. The Service shall inform the public through the sustained production and delivery of authoritative, timely, useful information about impacts on local, State, regional, tribal, national, and global scales. The service shall be user-centric, by ensuring that the information is accessible, consistent with users' ability to respond, and based on user needs and limitations.

[songwang41]

I encountered the similar with finetune with Fairseq version bart-large, https://github.com/pytorch/fairseq/issues/2347. Looks like it is a in complete sentence or starting from a middle of a sentence. It will be great if you shed some light on this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.